Oct 29 21:41:33.014203 ip-10-0-130-134 systemd[1]: Starting Kubernetes Kubelet... Oct 29 21:41:33.387006 ip-10-0-130-134 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:41:33.387006 ip-10-0-130-134 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 21:41:33.387006 ip-10-0-130-134 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:41:33.387006 ip-10-0-130-134 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:41:33.392198 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392119 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:33.392198 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392198 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:33.392198 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392202 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392205 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392209 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392212 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392214 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392217 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392220 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392223 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392225 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392228 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392231 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392234 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392237 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392240 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392250 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392255 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392259 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392262 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392265 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:33.392286 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392267 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392270 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392273 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392275 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392278 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392280 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392283 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392285 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392288 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392291 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392294 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392297 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392299 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392302 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392305 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392307 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392310 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392313 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392315 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:33.392726 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392317 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392321 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392325 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392328 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392331 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392334 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392336 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392340 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392342 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392345 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392348 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392350 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392352 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392355 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392358 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392361 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392364 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392367 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392369 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392372 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:33.393185 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392375 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392378 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392380 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392383 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392385 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392388 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392391 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392393 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392396 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392399 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392401 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392404 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392406 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392408 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392411 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392414 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392416 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392419 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392421 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:33.393654 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392424 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392427 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392429 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392433 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392436 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392438 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392441 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392443 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392447 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392450 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392452 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392455 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392458 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392460 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392463 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392465 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392468 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392470 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392473 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:33.394104 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392476 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392478 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392481 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392484 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392486 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392489 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392491 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.392495 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393316 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393324 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393328 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393330 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393333 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393336 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393339 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393342 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393345 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393348 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393350 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393353 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:33.394548 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393355 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393359 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393362 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393364 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393368 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393370 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393373 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393375 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393378 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393380 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393383 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393385 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393388 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393391 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393393 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393396 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393398 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393401 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393403 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:33.395076 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393406 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393408 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393411 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393414 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393416 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393419 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393422 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393426 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393428 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393431 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393434 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393436 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393439 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393441 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393445 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393447 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393450 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393452 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393455 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393458 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:33.395518 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393460 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393462 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393465 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393467 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393470 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393473 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393475 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393477 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393480 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393482 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393485 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393488 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393490 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393493 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393495 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393497 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393500 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393502 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393505 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:33.396005 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393510 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393513 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393516 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393520 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393522 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393525 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393527 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393530 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393533 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393535 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393537 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393540 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393542 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393545 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393547 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393550 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393552 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393554 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393557 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393560 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:33.396454 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393562 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393564 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393567 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393569 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393572 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393575 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393577 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393580 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393585 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393589 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393592 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393597 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393600 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393602 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393605 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.393607 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394108 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394114 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394118 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:33.396948 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394121 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394124 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394126 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394129 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394132 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394134 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394137 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394140 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394144 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394147 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394150 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394153 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394156 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394158 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394161 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394163 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394166 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394168 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394171 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394173 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:33.397396 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394176 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394178 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394181 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394183 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394188 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394190 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394193 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394195 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394198 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394201 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394205 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394208 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394211 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394214 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394217 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394219 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394222 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394225 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394227 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:33.397868 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394229 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394232 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394236 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394238 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394241 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394243 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394246 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394249 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394251 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394254 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394256 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394259 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394261 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394264 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394266 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394269 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394271 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394275 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394278 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:33.398330 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394281 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394283 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394286 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394288 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394291 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394294 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394298 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394301 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394303 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394306 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394309 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394311 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394314 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394316 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394319 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394322 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394324 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394327 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394329 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394332 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:33.398780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394335 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394337 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394340 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394342 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394345 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394348 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394351 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394353 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394355 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394358 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394362 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394364 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394367 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394370 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394372 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394375 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394377 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394381 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394385 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394388 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:33.399271 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394390 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394393 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394395 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394398 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.394401 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395509 2582 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395519 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395526 2582 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395531 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395535 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395539 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395543 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395548 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395551 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395554 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395558 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395561 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395564 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395567 2582 flags.go:64] FLAG: --cgroup-root="" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395570 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395573 2582 flags.go:64] FLAG: --client-ca-file="" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395576 2582 flags.go:64] FLAG: --cloud-provider="external" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395579 2582 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 21:41:33.399770 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395584 2582 flags.go:64] FLAG: --cluster-domain="" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395587 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395590 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395593 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395597 2582 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395601 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395604 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395608 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395612 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395615 2582 flags.go:64] FLAG: --contention-profiling="false" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395617 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395620 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395624 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395627 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395631 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395634 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395637 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395640 2582 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395643 2582 flags.go:64] FLAG: --enable-server="true" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395646 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395650 2582 flags.go:64] FLAG: --event-burst="100" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395653 2582 flags.go:64] FLAG: --event-qps="50" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395656 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395659 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 21:41:33.400322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395662 2582 flags.go:64] FLAG: --eviction-hard="" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395666 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395669 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395672 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395675 2582 flags.go:64] FLAG: --eviction-soft="" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395679 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395682 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395685 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395688 2582 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395691 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395694 2582 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395697 2582 flags.go:64] FLAG: --feature-gates="" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395701 2582 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395704 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395707 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395711 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395714 2582 flags.go:64] FLAG: --healthz-port="10248" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395718 2582 flags.go:64] FLAG: --help="false" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395721 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395724 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395727 2582 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395730 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395733 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 21:41:33.400922 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395737 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395740 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395742 2582 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395745 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395748 2582 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395752 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395755 2582 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395758 2582 flags.go:64] FLAG: --kube-reserved="" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395761 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395763 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395766 2582 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395769 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395772 2582 flags.go:64] FLAG: --lock-file="" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395775 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395778 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395781 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395787 2582 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395789 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395792 2582 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395795 2582 flags.go:64] FLAG: --logging-format="text" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395798 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395801 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395804 2582 flags.go:64] FLAG: --manifest-url="" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395807 2582 flags.go:64] FLAG: --manifest-url-header="" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395817 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 21:41:33.401466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395820 2582 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395825 2582 flags.go:64] FLAG: --max-pods="110" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395829 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395832 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395834 2582 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395837 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395840 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395843 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395846 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395853 2582 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395856 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395859 2582 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395862 2582 flags.go:64] FLAG: --pod-cidr="" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395865 2582 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395868 2582 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395871 2582 flags.go:64] FLAG: --pods-per-core="0" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395874 2582 flags.go:64] FLAG: --port="10250" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395877 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395880 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc7dd9c89c62e7d7" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395883 2582 flags.go:64] FLAG: --qos-reserved="" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395886 2582 flags.go:64] FLAG: --read-only-port="10255" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395889 2582 flags.go:64] FLAG: --register-node="true" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395892 2582 flags.go:64] FLAG: --register-with-taints="" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395896 2582 flags.go:64] FLAG: --registry-burst="10" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395898 2582 flags.go:64] FLAG: --registry-qps="5" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395911 2582 flags.go:64] FLAG: --reserved-cpus="" Oct 29 21:41:33.402073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395915 2582 flags.go:64] FLAG: --reserved-memory="" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395919 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395922 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395925 2582 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395928 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395931 2582 flags.go:64] FLAG: --runonce="false" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395934 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395937 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395941 2582 flags.go:64] FLAG: --seccomp-default="false" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395944 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395947 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395950 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395953 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395956 2582 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395959 2582 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395962 2582 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395964 2582 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395967 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395970 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395973 2582 flags.go:64] FLAG: --system-cgroups="" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395976 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395981 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395984 2582 flags.go:64] FLAG: --tls-cert-file="" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395987 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395991 2582 flags.go:64] FLAG: --tls-min-version="" Oct 29 21:41:33.402760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395994 2582 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.395997 2582 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.396000 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.396004 2582 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.396007 2582 flags.go:64] FLAG: --v="2" Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.396012 2582 flags.go:64] FLAG: --version="false" Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.396016 2582 flags.go:64] FLAG: --vmodule="" Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.396019 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.396023 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396168 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396172 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396176 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396183 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396186 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396189 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396192 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396195 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396198 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396201 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396204 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396206 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:33.403392 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396209 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396211 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396214 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396217 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396219 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396222 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396224 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396227 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396230 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396232 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396235 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396237 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396240 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396242 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396246 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396250 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396252 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396256 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396260 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:33.403896 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396263 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396266 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396269 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396271 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396275 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396278 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396280 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396283 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396286 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396289 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396292 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396294 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396297 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396300 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396302 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396305 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396307 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396310 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396312 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:33.404340 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396315 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396318 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396320 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396323 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396325 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396328 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396331 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396333 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396337 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396340 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396342 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396345 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396347 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396350 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396353 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396355 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396358 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396362 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396364 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396367 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:33.404815 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396370 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396372 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396375 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396378 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396381 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396384 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396386 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396389 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396391 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396394 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396396 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396399 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396401 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396404 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396407 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396409 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396412 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396414 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396417 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396419 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:33.405295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396423 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396426 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396429 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396431 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396434 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396436 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396439 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396441 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396444 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396448 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396450 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396453 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396456 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396459 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396462 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396464 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:33.405756 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.396492 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396527 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396532 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396535 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396538 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396549 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396552 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396555 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396558 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396561 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396564 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396566 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396569 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396572 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396574 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396577 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396582 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396584 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396587 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396590 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:33.406142 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396592 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396595 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396598 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396601 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396603 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396606 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396609 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396612 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396615 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396618 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396620 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396623 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396625 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396629 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396631 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396634 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396636 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396639 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396641 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:33.406612 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396644 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396646 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396649 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396651 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396654 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396657 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396661 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396664 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396667 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396672 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396675 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396679 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396682 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396684 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396687 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396689 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396692 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396694 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396697 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:33.407072 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396701 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396704 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396706 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396709 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396711 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396714 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396716 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396719 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396722 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396725 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396727 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396730 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396732 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396736 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396740 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396744 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396747 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396750 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396753 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:33.407505 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396757 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396759 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396762 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396765 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396768 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396771 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396773 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396776 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396778 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396781 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396783 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396786 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396788 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396792 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396795 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396798 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396800 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396803 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396806 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:33.407950 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396809 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396811 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396814 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396817 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396820 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396823 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396825 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396828 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396831 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396833 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.396836 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.404110 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.404125 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.404141 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.404146 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404224 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404229 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404233 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404236 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404239 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:33.408428 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404242 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404245 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404248 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404251 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404253 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404256 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404259 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404261 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404264 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404266 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404269 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404272 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404274 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404277 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404279 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404288 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404291 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404293 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404296 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:33.408917 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404299 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404302 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404304 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404307 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404310 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404313 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404316 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404319 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404322 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404324 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404327 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404329 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404332 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404335 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404337 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404340 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404343 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404346 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404349 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404351 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:33.409353 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404354 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404357 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404360 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404362 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404365 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404367 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404370 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404373 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404375 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404383 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404386 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404389 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404392 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404395 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404397 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404400 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404403 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404406 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404408 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404411 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:33.409808 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404413 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404416 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404418 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404421 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404424 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404427 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404429 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404432 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404435 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404438 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404442 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404445 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404448 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404451 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404453 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404456 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404458 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404461 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404463 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:33.410355 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404466 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404468 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404471 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404479 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404481 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404485 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404487 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404491 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404495 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404498 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404501 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404504 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404506 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404508 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404511 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404513 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404516 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404519 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404521 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404524 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:33.410791 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404526 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404529 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404531 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.404567 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404746 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404751 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404754 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404756 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404759 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404761 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404764 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404767 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404769 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404772 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404774 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404777 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:33.411268 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404785 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404789 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404792 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404794 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404796 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404800 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404803 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404805 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404808 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404810 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404813 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404815 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404818 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404821 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404823 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404826 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404829 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404831 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404834 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:33.411641 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404836 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404839 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404842 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404844 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404847 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404850 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404852 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404855 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404857 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404860 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404862 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404865 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404867 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404870 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404878 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404881 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404883 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404886 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404889 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:33.412092 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404891 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404894 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404897 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404899 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404917 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404920 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404923 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404925 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404928 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404930 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404932 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404935 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404938 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404940 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404943 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404946 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404948 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404951 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404955 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:33.412536 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404959 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404962 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404965 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404967 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404970 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404973 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404977 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404980 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404983 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404993 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404996 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.404999 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405001 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405004 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405006 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405009 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405012 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405014 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405017 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:33.413181 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405019 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405022 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405024 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405027 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405029 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405032 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405034 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405037 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405039 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405042 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405045 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405047 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405050 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405052 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405055 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405057 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405060 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:33.413925 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405063 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.405086 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405126 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405130 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405133 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405136 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405150 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405153 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405155 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405158 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405161 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405164 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405166 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405169 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405171 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:41:33.414419 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405174 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405176 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405179 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405181 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405183 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405186 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405188 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405191 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405193 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405196 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405198 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405201 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405203 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405206 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405208 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405211 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405213 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405216 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405218 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405221 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:41:33.414801 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405223 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405226 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405228 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405231 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405239 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405243 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405247 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405250 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405252 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405255 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405257 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405260 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405264 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405267 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405270 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405273 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405276 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405278 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405280 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:41:33.415362 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405283 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405285 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405288 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405290 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405293 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405295 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405298 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405301 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405304 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405307 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405309 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405312 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405314 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405316 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405319 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405322 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405324 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405327 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405336 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405339 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:41:33.415812 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405341 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405344 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405346 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405349 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405351 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405354 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405356 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405358 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405361 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405363 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405366 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405368 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405370 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405373 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405375 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405378 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405380 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405386 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405388 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405391 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:41:33.416315 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405393 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405397 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405399 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405402 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405404 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405407 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405409 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405411 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405414 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405416 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405419 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405435 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405438 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:33.405440 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.405496 2582 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.408854 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.409942 2582 server.go:1014] "Starting client certificate rotation" Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.410045 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 21:41:33.416858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.410076 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 21:41:33.417377 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.414489 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:41:33.418547 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.418530 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:41:33.429572 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.429553 2582 log.go:25] "Validated CRI v1 runtime API" Oct 29 21:41:33.432517 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.432500 2582 log.go:25] "Validated CRI v1 image API" Oct 29 21:41:33.433297 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.433269 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 21:41:33.438367 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.438343 2582 fs.go:135] Filesystem UUIDs: map[3e918413-df37-4169-9930-ad24e962ebf7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d2d80b37-64ed-4dd3-a50b-a65dac636611:/dev/nvme0n1p3] Oct 29 21:41:33.438433 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.438366 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 21:41:33.444274 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.444161 2582 manager.go:217] Machine: {Timestamp:2026-10-29 21:41:33.442654565 +0000 UTC m=+0.332301981 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100710 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29ed2a1f28fece6c94604c082cce54 SystemUUID:ec29ed2a-1f28-fece-6c94-604c082cce54 BootID:e432c050-bb27-4ac2-beeb-5ee9b17b693b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d2:d1:09:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d2:d1:09:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:8d:54:b3:d9:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 21:41:33.444274 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.444272 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 21:41:33.444391 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.444380 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 21:41:33.447817 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.447782 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 21:41:33.447960 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.447813 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 21:41:33.448021 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.447962 2582 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 21:41:33.448021 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.447971 2582 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 21:41:33.448021 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.448006 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:41:33.448104 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.448024 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:41:33.449228 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.449217 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 21:41:33.453724 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.453711 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 21:41:33.453855 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.453846 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 21:41:33.454773 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.454764 2582 kubelet.go:493] "Attempting to sync node with API server" Oct 29 21:41:33.454806 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.454778 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 21:41:33.454806 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.454788 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 21:41:33.454806 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.454797 2582 kubelet.go:405] "Adding apiserver pod source" Oct 29 21:41:33.454806 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.454805 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 21:41:33.455373 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.455352 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:41:33.460167 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.460141 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 21:41:33.467169 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.467152 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:41:33.468114 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.468100 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 21:41:33.468162 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.468131 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 21:41:33.468332 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.468316 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 21:41:33.468360 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.468341 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 21:41:33.468360 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.468348 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 21:41:33.468360 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.468353 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 21:41:33.468360 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.468359 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 21:41:33.468457 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.468365 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 21:41:33.469108 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.469099 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 21:41:33.469144 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.469109 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 21:41:33.469144 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.469116 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 21:41:33.469144 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.469122 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 21:41:33.469144 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.469131 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 21:41:33.469144 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.469140 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 21:41:33.469773 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.469765 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 21:41:33.469802 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.469776 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 21:41:33.473809 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.473795 2582 server.go:1263] "Started kubelet" Oct 29 21:41:33.473874 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.473847 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 21:41:33.474448 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.473885 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 21:41:33.474550 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.474473 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 21:41:33.474557 ip-10-0-130-134 systemd[1]: Started Kubernetes Kubelet. Oct 29 21:41:33.477103 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.477082 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 21:41:33.477335 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.477316 2582 server.go:317] "Adding debug handlers to kubelet server" Oct 29 21:41:33.480534 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.480449 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 21:41:33.482837 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.482818 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 21:41:33.483347 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.483332 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 21:41:33.483483 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.483460 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:33.484175 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484149 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 21:41:33.484289 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484270 2582 factory.go:55] Registering systemd factory Oct 29 21:41:33.484289 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484292 2582 factory.go:223] Registration of the systemd container factory successfully Oct 29 21:41:33.484415 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484280 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 21:41:33.484415 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484170 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 21:41:33.484489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484445 2582 reconstruct.go:97] "Volume reconstruction finished" Oct 29 21:41:33.484489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484454 2582 reconciler.go:29] "Reconciler: start to sync state" Oct 29 21:41:33.484560 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484532 2582 factory.go:153] Registering CRI-O factory Oct 29 21:41:33.484560 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484557 2582 factory.go:223] Registration of the crio container factory successfully Oct 29 21:41:33.484613 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484606 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 21:41:33.484651 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484623 2582 factory.go:103] Registering Raw factory Oct 29 21:41:33.484651 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.484633 2582 manager.go:1199] Started watching for new ooms in manager Oct 29 21:41:33.485699 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.485682 2582 manager.go:319] Starting recovery of all containers Oct 29 21:41:33.500084 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.500067 2582 manager.go:324] Recovery completed Oct 29 21:41:33.503215 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.503194 2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 29 21:41:33.505073 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.505039 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 21:41:33.505073 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.505022 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-134.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 21:41:33.506334 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.505044 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-134.ec2.internal.18e31e0755a05d49 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-134.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-134.ec2.internal,},FirstTimestamp:2026-10-29 21:41:33.473766729 +0000 UTC m=+0.363414144,LastTimestamp:2026-10-29 21:41:33.473766729 +0000 UTC m=+0.363414144,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-134.ec2.internal,}" Oct 29 21:41:33.506428 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.506417 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:33.507717 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.507694 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 21:41:33.508636 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.508613 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:33.508716 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.508651 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:33.508716 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.508666 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:33.509259 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.509245 2582 cpu_manager.go:226] "Starting" policy="none" Oct 29 21:41:33.509259 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.509259 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 21:41:33.509367 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.509277 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 21:41:33.509367 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.509283 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:33.511067 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.511054 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:33.511109 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.511079 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:33.511109 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.511091 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:33.511404 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.511391 2582 policy_none.go:50] "Start" Oct 29 21:41:33.511433 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.511409 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 21:41:33.511433 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.511422 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 21:41:33.513373 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.513361 2582 policy_none.go:44] "Start" Oct 29 21:41:33.524111 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.524035 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-134.ec2.internal.18e31e0757b46898 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-134.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-134.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-134.ec2.internal,},FirstTimestamp:2026-10-29 21:41:33.508634776 +0000 UTC m=+0.398282197,LastTimestamp:2026-10-29 21:41:33.508634776 +0000 UTC m=+0.398282197,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-134.ec2.internal,}" Oct 29 21:41:33.524180 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.524088 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 29 21:41:33.524495 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.524475 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 21:41:33.537357 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.537334 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 29 21:41:33.546125 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.546108 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xbgf7" Oct 29 21:41:33.548452 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.548430 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 21:41:33.550835 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.550821 2582 manager.go:341] "Starting Device Plugin manager" Oct 29 21:41:33.550919 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.550854 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 21:41:33.550919 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.550865 2582 server.go:87] "Starting device plugin registration server" Oct 29 21:41:33.551127 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.551114 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 21:41:33.551192 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.551128 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 21:41:33.551240 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.551219 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 21:41:33.551371 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.551321 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 21:41:33.551371 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.551333 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 21:41:33.551948 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.551929 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 21:41:33.552008 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.551977 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:33.556896 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.556829 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-134.ec2.internal.18e31e0757b4c5e7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-134.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-134.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-134.ec2.internal,},FirstTimestamp:2026-10-29 21:41:33.508658663 +0000 UTC m=+0.398306078,LastTimestamp:2026-10-29 21:41:33.508658663 +0000 UTC m=+0.398306078,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-134.ec2.internal,}" Oct 29 21:41:33.558617 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.558602 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xbgf7" Oct 29 21:41:33.572003 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.571932 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-134.ec2.internal.18e31e0757b50294 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-134.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-134.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-134.ec2.internal,},FirstTimestamp:2026-10-29 21:41:33.508674196 +0000 UTC m=+0.398321612,LastTimestamp:2026-10-29 21:41:33.508674196 +0000 UTC m=+0.398321612,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-134.ec2.internal,}" Oct 29 21:41:33.614251 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.614222 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 21:41:33.616125 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.615542 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 21:41:33.616125 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.615563 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 21:41:33.616125 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.615585 2582 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 21:41:33.616125 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.615624 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 29 21:41:33.632599 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.632579 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:33.651501 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.651452 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:33.652243 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.652228 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:33.652325 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.652258 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:33.652325 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.652268 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:33.652325 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.652290 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.673652 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.673635 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.673703 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.673654 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-134.ec2.internal\": node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:33.713126 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.713105 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:33.716181 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.716165 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal"] Oct 29 21:41:33.716245 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.716214 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:33.716991 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.716976 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:33.717070 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.717003 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:33.717070 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.717012 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:33.719231 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.719219 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:33.719380 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.719367 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.719430 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.719395 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:33.719933 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.719918 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:33.720003 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.719947 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:33.720003 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.719957 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:33.720003 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.719918 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:33.720095 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.720025 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:33.720095 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.720036 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:33.721982 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.721969 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.722033 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.721992 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:33.722715 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.722698 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:33.722798 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.722727 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:33.722798 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.722740 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:33.750244 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.750228 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-134.ec2.internal\" not found" node="ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.752411 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.752395 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-134.ec2.internal\" not found" node="ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.786580 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.786558 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.786658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.786586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.786658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.786603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3439c72aff40b3d1f617a398c69856e9-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" (UID: \"3439c72aff40b3d1f617a398c69856e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.813567 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.813549 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:33.887566 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.887533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.887566 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.887566 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.887749 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.887585 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3439c72aff40b3d1f617a398c69856e9-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" (UID: \"3439c72aff40b3d1f617a398c69856e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.887749 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.887614 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.887749 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.887616 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.887749 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:33.887645 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3439c72aff40b3d1f617a398c69856e9-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" (UID: \"3439c72aff40b3d1f617a398c69856e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:41:33.914188 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:33.914138 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:34.015268 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:34.015101 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:34.053788 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.053767 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:41:34.055913 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.055892 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:41:34.115995 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:34.115965 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:34.216529 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:34.216462 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:34.317173 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:34.317143 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:34.409635 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.409599 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 21:41:34.410171 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.409778 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 21:41:34.417446 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:34.417417 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:34.518416 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:34.518391 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:34.563895 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.563841 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:36:34 +0000 UTC" deadline="2028-07-30 23:10:14.826974956 +0000 UTC" Oct 29 21:41:34.563895 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.563892 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15361h28m40.26308822s" Oct 29 21:41:34.599668 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.599634 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:34.618564 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:34.618530 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:34.627436 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.627410 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:34.642295 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:34.642259 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05887799647c9e22c760e149ed245f3e.slice/crio-44f6a8de86887cbdb030477928904527a0b60058ca463e571005572d449fa744 WatchSource:0}: Error finding container 44f6a8de86887cbdb030477928904527a0b60058ca463e571005572d449fa744: Status 404 returned error can't find the container with id 44f6a8de86887cbdb030477928904527a0b60058ca463e571005572d449fa744 Oct 29 21:41:34.649265 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.649241 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:41:34.652637 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:34.652611 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3439c72aff40b3d1f617a398c69856e9.slice/crio-f4357f1988c399aa3fb5b008a5f7c483767e42a0414013edf71baa45ca89b904 WatchSource:0}: Error finding container f4357f1988c399aa3fb5b008a5f7c483767e42a0414013edf71baa45ca89b904: Status 404 returned error can't find the container with id f4357f1988c399aa3fb5b008a5f7c483767e42a0414013edf71baa45ca89b904 Oct 29 21:41:34.718657 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:34.718620 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:34.719784 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.719767 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:41:34.721161 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.721141 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:41:34.721285 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.721174 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:41:34.721285 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.721184 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:41:34.820135 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:34.820060 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:41:34.862036 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.862008 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:34.884724 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.884698 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:41:34.910340 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.910321 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:41:34.911464 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.911451 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:41:34.940835 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:34.940811 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:41:35.456100 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.456067 2582 apiserver.go:52] "Watching apiserver" Oct 29 21:41:35.483648 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.483615 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 21:41:35.506755 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.506718 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 21:41:35.507493 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.507112 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-s8t9b","openshift-multus/network-metrics-daemon-rcs9l","openshift-multus/multus-additional-cni-plugins-mdxzj","openshift-network-diagnostics/network-check-target-289sc","openshift-ovn-kubernetes/ovnkube-node-hvpq9","kube-system/global-pull-secret-syncer-29g9k","kube-system/konnectivity-agent-krcw7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd","openshift-image-registry/node-ca-tjgwl","openshift-cluster-node-tuning-operator/tuned-f7mnz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal","openshift-network-operator/iptables-alerter-2s6z8","kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal","openshift-dns/node-resolver-t6k4p"] Oct 29 21:41:35.509926 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.509895 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:35.512518 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.512500 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.515779 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.515756 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.521766 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.521744 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:35.521878 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.521838 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:35.525778 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.525757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.528082 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.528060 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:35.528180 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.528151 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:35.530803 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.530787 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.533661 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.533642 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.535824 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.535792 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 21:41:35.535931 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.535878 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 21:41:35.536415 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.535976 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 21:41:35.536415 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.536021 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mldf8\"" Oct 29 21:41:35.536415 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.536045 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 21:41:35.536415 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.536072 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 21:41:35.536415 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.536123 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 21:41:35.536415 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.536129 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 21:41:35.536415 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.536185 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nll6c\"" Oct 29 21:41:35.536415 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.536279 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:35.536415 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.536331 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:35.537777 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.537545 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 21:41:35.537777 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.537719 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 21:41:35.537777 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.537737 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6x2hg\"" Oct 29 21:41:35.537969 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.537808 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 21:41:35.542691 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.540393 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.542691 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.541404 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 21:41:35.544643 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.544625 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.548368 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.548345 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.552367 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.552348 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pfgv8" Oct 29 21:41:35.574657 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.574492 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pfgv8" Oct 29 21:41:35.585518 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.585502 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 21:41:35.597208 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-conf\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.597346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597224 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.597346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597255 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-bin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.597346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89de847b-bcca-4f80-af6b-071681d3afdb-serviceca\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.597526 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597353 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysconfig\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.597526 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-run\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.597526 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597429 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-system-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.597526 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597461 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cnibin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.597526 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-multus\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.597526 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-etc-kubernetes\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.597526 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597511 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzw9z\" (UniqueName: \"kubernetes.io/projected/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kube-api-access-rzw9z\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597545 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597567 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-kubelet\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/587e4eb6-4e54-4c6e-a640-71976f13827c-ovn-node-metrics-cert\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597598 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-multus-certs\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597614 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73219cee-7208-4f68-ad0b-bc9cb0bcb199-agent-certs\") pod \"konnectivity-agent-krcw7\" (UID: \"73219cee-7208-4f68-ad0b-bc9cb0bcb199\") " pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597629 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-device-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597659 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-var-lib-kubelet\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdmgp\" (UniqueName: \"kubernetes.io/projected/88711e24-d089-4fbb-9b36-fb015fe7f9c9-kube-api-access-bdmgp\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-netd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597743 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597761 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597781 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-daemon-config\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597800 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84058be2-867e-4c5a-a614-bccfd7af96fa-hosts-file\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84058be2-867e-4c5a-a614-bccfd7af96fa-tmp-dir\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.597858 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597848 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4eb95095-a274-43e2-adbd-d88824033bb1-iptables-alerter-script\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597863 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-cnibin\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-netns\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597904 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-netns\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597924 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-systemd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-node-log\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.597998 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn95n\" (UniqueName: \"kubernetes.io/projected/89de847b-bcca-4f80-af6b-071681d3afdb-kube-api-access-bn95n\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-host\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-env-overrides\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598099 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhmp9\" (UniqueName: \"kubernetes.io/projected/97611cb7-e267-4d11-b20a-c52d833cdefc-kube-api-access-bhmp9\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-kubelet\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598155 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-hostroot\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73219cee-7208-4f68-ad0b-bc9cb0bcb199-konnectivity-ca\") pod \"konnectivity-agent-krcw7\" (UID: \"73219cee-7208-4f68-ad0b-bc9cb0bcb199\") " pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nfl\" (UniqueName: \"kubernetes.io/projected/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-kube-api-access-q2nfl\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:35.598658 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598218 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-modprobe-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598233 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cni-binary-copy\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598251 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjpx\" (UniqueName: \"kubernetes.io/projected/84058be2-867e-4c5a-a614-bccfd7af96fa-kube-api-access-6kjpx\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9hc6\" (UniqueName: \"kubernetes.io/projected/587e4eb6-4e54-4c6e-a640-71976f13827c-kube-api-access-r9hc6\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598283 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598297 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89de847b-bcca-4f80-af6b-071681d3afdb-host\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598321 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-slash\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598337 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-sys\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-socket-dir-parent\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598400 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-conf-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-sys-fs\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-dbus\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-var-lib-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-systemd\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.599489 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-tuned\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598492 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-etc-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598508 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-k8s-cni-cncf-io\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598524 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb95095-a274-43e2-adbd-d88824033bb1-host-slash\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598619 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598658 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-script-lib\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598674 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-kubelet-config\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598689 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-tmp\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598705 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr9fw\" (UniqueName: \"kubernetes.io/projected/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-kube-api-access-qr9fw\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtcq\" (UniqueName: \"kubernetes.io/projected/4eb95095-a274-43e2-adbd-d88824033bb1-kube-api-access-mgtcq\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598735 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-ovn\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-config\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-socket-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-registration-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-systemd-units\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.600342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-system-cni-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.601065 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598844 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.601065 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598862 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-log-socket\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.601065 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-bin\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.601065 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598892 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-lib-modules\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.601065 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598909 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.601065 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.598927 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-kubernetes\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.621398 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.621339 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" event={"ID":"3439c72aff40b3d1f617a398c69856e9","Type":"ContainerStarted","Data":"f4357f1988c399aa3fb5b008a5f7c483767e42a0414013edf71baa45ca89b904"} Oct 29 21:41:35.622803 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.622776 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" event={"ID":"05887799647c9e22c760e149ed245f3e","Type":"ContainerStarted","Data":"44f6a8de86887cbdb030477928904527a0b60058ca463e571005572d449fa744"} Oct 29 21:41:35.641269 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641241 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 21:41:35.641269 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641241 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 21:41:35.641269 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641253 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 21:41:35.641549 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641253 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 21:41:35.641549 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641270 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:41:35.641715 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641699 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 21:41:35.641715 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641706 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 21:41:35.641833 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641721 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 21:41:35.641833 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641814 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 21:41:35.641939 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641854 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 21:41:35.641939 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641885 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 21:41:35.641939 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641885 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:41:35.642133 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.641977 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 21:41:35.642133 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.642028 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 21:41:35.656067 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.656042 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 21:41:35.656233 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.656073 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 21:41:35.656233 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.656085 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rbnwt\"" Oct 29 21:41:35.656233 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.656093 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fcqkf\"" Oct 29 21:41:35.656233 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.656100 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2qkvr\"" Oct 29 21:41:35.656233 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.656123 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cndgh\"" Oct 29 21:41:35.656233 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.656133 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zsmh\"" Oct 29 21:41:35.656233 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.656144 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4df8x\"" Oct 29 21:41:35.656233 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.656172 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699778 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-conf-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-sys-fs\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-dbus\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699866 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-conf-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-var-lib-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699930 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-systemd\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-tuned\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699975 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-sys-fs\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699987 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-systemd\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699941 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-var-lib-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.699993 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-etc-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-k8s-cni-cncf-io\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb95095-a274-43e2-adbd-d88824033bb1-host-slash\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700071 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-k8s-cni-cncf-io\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700076 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-dbus\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700093 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.706575 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700119 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb95095-a274-43e2-adbd-d88824033bb1-host-slash\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700164 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700036 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-etc-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700192 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-script-lib\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-kubelet-config\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700244 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-tmp\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700273 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700296 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-kubelet-config\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700289 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr9fw\" (UniqueName: \"kubernetes.io/projected/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-kube-api-access-qr9fw\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtcq\" (UniqueName: \"kubernetes.io/projected/4eb95095-a274-43e2-adbd-d88824033bb1-kube-api-access-mgtcq\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700386 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-ovn\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700413 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-config\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-socket-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-registration-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700499 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-systemd-units\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707302 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-system-cni-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-log-socket\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700595 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-bin\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700622 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-lib-modules\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700646 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-kubernetes\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-conf\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700688 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-registration-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700716 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700736 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-script-lib\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700753 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-log-socket\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700779 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-systemd-units\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700804 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-bin\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700828 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-system-cni-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700877 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-lib-modules\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700879 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-ovn\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.707908 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700977 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-kubernetes\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.700985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-bin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701005 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-socket-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89de847b-bcca-4f80-af6b-071681d3afdb-serviceca\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701042 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysconfig\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701047 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701067 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-run\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701086 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-conf\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701092 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-system-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701092 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-bin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701153 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-run\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701162 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-config\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701137 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysconfig\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701207 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-system-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701217 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cnibin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701243 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-multus\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701264 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-etc-kubernetes\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.708535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701276 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cnibin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzw9z\" (UniqueName: \"kubernetes.io/projected/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kube-api-access-rzw9z\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701360 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-kubelet\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701388 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/587e4eb6-4e54-4c6e-a640-71976f13827c-ovn-node-metrics-cert\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-multus-certs\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73219cee-7208-4f68-ad0b-bc9cb0bcb199-agent-certs\") pod \"konnectivity-agent-krcw7\" (UID: \"73219cee-7208-4f68-ad0b-bc9cb0bcb199\") " pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701452 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89de847b-bcca-4f80-af6b-071681d3afdb-serviceca\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701492 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-etc-kubernetes\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701496 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-multus\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-device-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-var-lib-kubelet\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701540 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-device-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701559 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmgp\" (UniqueName: \"kubernetes.io/projected/88711e24-d089-4fbb-9b36-fb015fe7f9c9-kube-api-access-bdmgp\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701582 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-netd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701592 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-multus-certs\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.709164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701629 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701647 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701656 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701652 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-daemon-config\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701700 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84058be2-867e-4c5a-a614-bccfd7af96fa-hosts-file\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701703 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-var-lib-kubelet\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84058be2-867e-4c5a-a614-bccfd7af96fa-tmp-dir\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701753 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4eb95095-a274-43e2-adbd-d88824033bb1-iptables-alerter-script\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701778 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-cnibin\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-netns\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701827 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-netns\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701830 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-systemd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701877 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-node-log\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn95n\" (UniqueName: \"kubernetes.io/projected/89de847b-bcca-4f80-af6b-071681d3afdb-kube-api-access-bn95n\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.709997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701951 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-host\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-env-overrides\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702552 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-daemon-config\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.701702 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-netd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702555 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmp9\" (UniqueName: \"kubernetes.io/projected/97611cb7-e267-4d11-b20a-c52d833cdefc-kube-api-access-bhmp9\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-kubelet\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702820 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-hostroot\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73219cee-7208-4f68-ad0b-bc9cb0bcb199-konnectivity-ca\") pod \"konnectivity-agent-krcw7\" (UID: \"73219cee-7208-4f68-ad0b-bc9cb0bcb199\") " pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nfl\" (UniqueName: \"kubernetes.io/projected/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-kube-api-access-q2nfl\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-modprobe-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cni-binary-copy\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.702960 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjpx\" (UniqueName: \"kubernetes.io/projected/84058be2-867e-4c5a-a614-bccfd7af96fa-kube-api-access-6kjpx\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703012 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9hc6\" (UniqueName: \"kubernetes.io/projected/587e4eb6-4e54-4c6e-a640-71976f13827c-kube-api-access-r9hc6\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703040 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89de847b-bcca-4f80-af6b-071681d3afdb-host\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703093 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-slash\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.710682 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703106 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4eb95095-a274-43e2-adbd-d88824033bb1-iptables-alerter-script\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703119 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703124 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703146 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-sys\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-socket-dir-parent\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703296 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-socket-dir-parent\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703348 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84058be2-867e-4c5a-a614-bccfd7af96fa-hosts-file\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703376 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-cnibin\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703425 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-netns\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703467 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-netns\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703487 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-env-overrides\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703508 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-systemd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703559 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-host\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.703582 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703597 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84058be2-867e-4c5a-a614-bccfd7af96fa-tmp-dir\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.703644 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:36.203625146 +0000 UTC m=+3.093272573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703659 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-kubelet\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.711414 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703697 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-hostroot\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703773 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89de847b-bcca-4f80-af6b-071681d3afdb-host\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.704072 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-modprobe-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.704224 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73219cee-7208-4f68-ad0b-bc9cb0bcb199-konnectivity-ca\") pod \"konnectivity-agent-krcw7\" (UID: \"73219cee-7208-4f68-ad0b-bc9cb0bcb199\") " pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.703063 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.704299 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-slash\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.704321 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.704368 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret podName:fb5b0077-ddac-46d3-ba06-5fc065be5b53 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:36.204353873 +0000 UTC m=+3.094001293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret") pod "global-pull-secret-syncer-29g9k" (UID: "fb5b0077-ddac-46d3-ba06-5fc065be5b53") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.704399 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.704410 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-kubelet\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.704440 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-node-log\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.704465 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-sys\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.704646 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cni-binary-copy\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.704916 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73219cee-7208-4f68-ad0b-bc9cb0bcb199-agent-certs\") pod \"konnectivity-agent-krcw7\" (UID: \"73219cee-7208-4f68-ad0b-bc9cb0bcb199\") " pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.704993 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.705545 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-tmp\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.706711 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-tuned\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.712030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.707690 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/587e4eb6-4e54-4c6e-a640-71976f13827c-ovn-node-metrics-cert\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.728835 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.728801 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtcq\" (UniqueName: \"kubernetes.io/projected/4eb95095-a274-43e2-adbd-d88824033bb1-kube-api-access-mgtcq\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.729057 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.729034 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmgp\" (UniqueName: \"kubernetes.io/projected/88711e24-d089-4fbb-9b36-fb015fe7f9c9-kube-api-access-bdmgp\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.730799 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.730773 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzw9z\" (UniqueName: \"kubernetes.io/projected/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kube-api-access-rzw9z\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.731327 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.731283 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr9fw\" (UniqueName: \"kubernetes.io/projected/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-kube-api-access-qr9fw\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.731434 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.731422 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:35.731508 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.731441 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:35.731508 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.731455 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zknj5 for pod openshift-network-diagnostics/network-check-target-289sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:35.731589 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:35.731515 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5 podName:3dbda061-df54-482d-96c6-851f1acf58a4 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:36.231495188 +0000 UTC m=+3.121142607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zknj5" (UniqueName: "kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5") pod "network-check-target-289sc" (UID: "3dbda061-df54-482d-96c6-851f1acf58a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:35.733748 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.733720 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn95n\" (UniqueName: \"kubernetes.io/projected/89de847b-bcca-4f80-af6b-071681d3afdb-kube-api-access-bn95n\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.734465 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.734440 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9hc6\" (UniqueName: \"kubernetes.io/projected/587e4eb6-4e54-4c6e-a640-71976f13827c-kube-api-access-r9hc6\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.735079 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.735057 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nfl\" (UniqueName: \"kubernetes.io/projected/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-kube-api-access-q2nfl\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:35.735677 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.735656 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmp9\" (UniqueName: \"kubernetes.io/projected/97611cb7-e267-4d11-b20a-c52d833cdefc-kube-api-access-bhmp9\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.739471 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.739355 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjpx\" (UniqueName: \"kubernetes.io/projected/84058be2-867e-4c5a-a614-bccfd7af96fa-kube-api-access-6kjpx\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.760780 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.760752 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:41:35.830608 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.830573 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:35.842802 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:35.842762 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73219cee_7208_4f68_ad0b_bc9cb0bcb199.slice/crio-71c090c7e7270dc7cc8a18b9826c3d50fcfa30d9634863c2f555d2c61f057f15 WatchSource:0}: Error finding container 71c090c7e7270dc7cc8a18b9826c3d50fcfa30d9634863c2f555d2c61f057f15: Status 404 returned error can't find the container with id 71c090c7e7270dc7cc8a18b9826c3d50fcfa30d9634863c2f555d2c61f057f15 Oct 29 21:41:35.849938 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.849916 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:41:35.859902 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:35.859872 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf1d737_3cce_4759_8bde_21c7e0dff0ae.slice/crio-9371a4bc96e83be171ae0fdacfe6791bde2e3aedc040569459d08e98062da97e WatchSource:0}: Error finding container 9371a4bc96e83be171ae0fdacfe6791bde2e3aedc040569459d08e98062da97e: Status 404 returned error can't find the container with id 9371a4bc96e83be171ae0fdacfe6791bde2e3aedc040569459d08e98062da97e Oct 29 21:41:35.872323 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.872297 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:41:35.875193 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.875160 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:41:35.877608 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.877574 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s8t9b" Oct 29 21:41:35.883577 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:35.883542 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89de847b_bcca_4f80_af6b_071681d3afdb.slice/crio-0024dfc8c86490a909c81911a36e9e55a1b30073969387326eb0b7f225852080 WatchSource:0}: Error finding container 0024dfc8c86490a909c81911a36e9e55a1b30073969387326eb0b7f225852080: Status 404 returned error can't find the container with id 0024dfc8c86490a909c81911a36e9e55a1b30073969387326eb0b7f225852080 Oct 29 21:41:35.887282 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.887108 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:41:35.887634 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:35.887591 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97611cb7_e267_4d11_b20a_c52d833cdefc.slice/crio-7a6f507bbd09eb4724da3ed66ffd47c18814cc9e145f2c5ecbc8bcd7c7224fe8 WatchSource:0}: Error finding container 7a6f507bbd09eb4724da3ed66ffd47c18814cc9e145f2c5ecbc8bcd7c7224fe8: Status 404 returned error can't find the container with id 7a6f507bbd09eb4724da3ed66ffd47c18814cc9e145f2c5ecbc8bcd7c7224fe8 Oct 29 21:41:35.893181 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.893143 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:35.895939 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.895673 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:41:35.898838 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:35.898803 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88711e24_d089_4fbb_9b36_fb015fe7f9c9.slice/crio-e3b0d33ddcca32a49ea7259e916e9144a0b3dfbe6ff373a1ce90e985db448499 WatchSource:0}: Error finding container e3b0d33ddcca32a49ea7259e916e9144a0b3dfbe6ff373a1ce90e985db448499: Status 404 returned error can't find the container with id e3b0d33ddcca32a49ea7259e916e9144a0b3dfbe6ff373a1ce90e985db448499 Oct 29 21:41:35.899773 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:35.899755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:41:35.909241 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:35.909213 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587e4eb6_4e54_4c6e_a640_71976f13827c.slice/crio-e81dd74f03f2acbdef9d82f844a6d94f8fe760bcde0c91a1cfbe2853d6c1c020 WatchSource:0}: Error finding container e81dd74f03f2acbdef9d82f844a6d94f8fe760bcde0c91a1cfbe2853d6c1c020: Status 404 returned error can't find the container with id e81dd74f03f2acbdef9d82f844a6d94f8fe760bcde0c91a1cfbe2853d6c1c020 Oct 29 21:41:35.913101 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:35.913075 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84058be2_867e_4c5a_a614_bccfd7af96fa.slice/crio-737395ecc4418578cf52b2ba4f18a331d771cbc7a0d576db82815abf68e66f6b WatchSource:0}: Error finding container 737395ecc4418578cf52b2ba4f18a331d771cbc7a0d576db82815abf68e66f6b: Status 404 returned error can't find the container with id 737395ecc4418578cf52b2ba4f18a331d771cbc7a0d576db82815abf68e66f6b Oct 29 21:41:35.916128 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:41:35.916102 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb95095_a274_43e2_adbd_d88824033bb1.slice/crio-0eda55ea5d4453f536acf3538eac8378674a1e52ee5ae524cb7bfdb62b528a10 WatchSource:0}: Error finding container 0eda55ea5d4453f536acf3538eac8378674a1e52ee5ae524cb7bfdb62b528a10: Status 404 returned error can't find the container with id 0eda55ea5d4453f536acf3538eac8378674a1e52ee5ae524cb7bfdb62b528a10 Oct 29 21:41:36.206157 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.206125 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:36.206291 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.206229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:36.206350 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:36.206303 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:36.206350 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:36.206332 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:36.206421 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:36.206378 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret podName:fb5b0077-ddac-46d3-ba06-5fc065be5b53 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:37.20635791 +0000 UTC m=+4.096005328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret") pod "global-pull-secret-syncer-29g9k" (UID: "fb5b0077-ddac-46d3-ba06-5fc065be5b53") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:36.206421 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:36.206400 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:37.206387792 +0000 UTC m=+4.096035200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:36.307069 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.307038 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:36.307203 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:36.307162 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:36.307203 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:36.307176 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:36.307203 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:36.307186 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zknj5 for pod openshift-network-diagnostics/network-check-target-289sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:36.307378 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:36.307238 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5 podName:3dbda061-df54-482d-96c6-851f1acf58a4 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:37.307225636 +0000 UTC m=+4.196873054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zknj5" (UniqueName: "kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5") pod "network-check-target-289sc" (UID: "3dbda061-df54-482d-96c6-851f1acf58a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:36.578215 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.577825 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:36:36 +0000 UTC" deadline="2028-04-14 03:47:10.443183778 +0000 UTC" Oct 29 21:41:36.578215 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.578134 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12774h5m33.865056116s" Oct 29 21:41:36.628540 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.628392 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2s6z8" event={"ID":"4eb95095-a274-43e2-adbd-d88824033bb1","Type":"ContainerStarted","Data":"0eda55ea5d4453f536acf3538eac8378674a1e52ee5ae524cb7bfdb62b528a10"} Oct 29 21:41:36.634880 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.634847 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" event={"ID":"97611cb7-e267-4d11-b20a-c52d833cdefc","Type":"ContainerStarted","Data":"7a6f507bbd09eb4724da3ed66ffd47c18814cc9e145f2c5ecbc8bcd7c7224fe8"} Oct 29 21:41:36.645281 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.645201 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t6k4p" event={"ID":"84058be2-867e-4c5a-a614-bccfd7af96fa","Type":"ContainerStarted","Data":"737395ecc4418578cf52b2ba4f18a331d771cbc7a0d576db82815abf68e66f6b"} Oct 29 21:41:36.649394 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.649081 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" event={"ID":"587e4eb6-4e54-4c6e-a640-71976f13827c","Type":"ContainerStarted","Data":"e81dd74f03f2acbdef9d82f844a6d94f8fe760bcde0c91a1cfbe2853d6c1c020"} Oct 29 21:41:36.652418 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.652392 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" event={"ID":"88711e24-d089-4fbb-9b36-fb015fe7f9c9","Type":"ContainerStarted","Data":"e3b0d33ddcca32a49ea7259e916e9144a0b3dfbe6ff373a1ce90e985db448499"} Oct 29 21:41:36.657269 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.657232 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s8t9b" event={"ID":"1a8cd583-ba4f-4f3a-9f42-f43d63462d32","Type":"ContainerStarted","Data":"6d61eeb039f80edf4e1629e9d376fcc83534a34d8e011f58f3225cf5ae27994a"} Oct 29 21:41:36.661188 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.661160 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tjgwl" event={"ID":"89de847b-bcca-4f80-af6b-071681d3afdb","Type":"ContainerStarted","Data":"0024dfc8c86490a909c81911a36e9e55a1b30073969387326eb0b7f225852080"} Oct 29 21:41:36.666248 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.666223 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" event={"ID":"dcf1d737-3cce-4759-8bde-21c7e0dff0ae","Type":"ContainerStarted","Data":"9371a4bc96e83be171ae0fdacfe6791bde2e3aedc040569459d08e98062da97e"} Oct 29 21:41:36.669500 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.669463 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-krcw7" event={"ID":"73219cee-7208-4f68-ad0b-bc9cb0bcb199","Type":"ContainerStarted","Data":"71c090c7e7270dc7cc8a18b9826c3d50fcfa30d9634863c2f555d2c61f057f15"} Oct 29 21:41:36.674614 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.674585 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" event={"ID":"3439c72aff40b3d1f617a398c69856e9","Type":"ContainerStarted","Data":"914f02d6c929974c682b3727af98ff24d228bcbaf94860a4f8bc6f7a386e0285"} Oct 29 21:41:36.721286 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:36.721221 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" podStartSLOduration=1.72120244 podStartE2EDuration="1.72120244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:36.720857376 +0000 UTC m=+3.610504803" watchObservedRunningTime="2026-10-29 21:41:36.72120244 +0000 UTC m=+3.610849869" Oct 29 21:41:37.214523 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:37.214475 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:37.214705 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:37.214545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:37.214705 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.214684 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:37.214830 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.214748 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret podName:fb5b0077-ddac-46d3-ba06-5fc065be5b53 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:39.214728236 +0000 UTC m=+6.104375658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret") pod "global-pull-secret-syncer-29g9k" (UID: "fb5b0077-ddac-46d3-ba06-5fc065be5b53") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:37.215205 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.215185 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:37.215292 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.215242 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:39.215226036 +0000 UTC m=+6.104873443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:37.315998 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:37.315956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:37.316216 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.316197 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:37.316288 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.316225 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:37.316288 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.316238 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zknj5 for pod openshift-network-diagnostics/network-check-target-289sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:37.316404 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.316299 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5 podName:3dbda061-df54-482d-96c6-851f1acf58a4 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:39.316280978 +0000 UTC m=+6.205928385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zknj5" (UniqueName: "kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5") pod "network-check-target-289sc" (UID: "3dbda061-df54-482d-96c6-851f1acf58a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:37.579386 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:37.579260 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:36:36 +0000 UTC" deadline="2028-07-19 08:44:34.497334096 +0000 UTC" Oct 29 21:41:37.579386 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:37.579311 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15083h2m56.918026131s" Oct 29 21:41:37.619444 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:37.618645 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:37.619444 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.618778 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:37.619444 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:37.619111 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:37.619444 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.619229 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:37.619444 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:37.619298 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:37.619444 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:37.619367 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:37.692247 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:37.692199 2582 generic.go:356] "Generic (PLEG): container finished" podID="05887799647c9e22c760e149ed245f3e" containerID="08886d7a4243fc68c8749fd96195f4086600663d8bf928299ef03fc8c16de61a" exitCode=0 Oct 29 21:41:37.692436 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:37.692276 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" event={"ID":"05887799647c9e22c760e149ed245f3e","Type":"ContainerDied","Data":"08886d7a4243fc68c8749fd96195f4086600663d8bf928299ef03fc8c16de61a"} Oct 29 21:41:38.722311 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:38.722231 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" event={"ID":"05887799647c9e22c760e149ed245f3e","Type":"ContainerStarted","Data":"45b2a7d259146657524cc64342addc00a1568c2f85cb3ada92a922e3f10baf90"} Oct 29 21:41:39.236542 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:39.236461 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:39.236542 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:39.236525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:39.236791 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.236673 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:39.236791 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.236740 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret podName:fb5b0077-ddac-46d3-ba06-5fc065be5b53 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:43.236721843 +0000 UTC m=+10.126369266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret") pod "global-pull-secret-syncer-29g9k" (UID: "fb5b0077-ddac-46d3-ba06-5fc065be5b53") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:39.237155 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.237135 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:39.237248 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.237208 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:43.237190306 +0000 UTC m=+10.126837717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:39.337535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:39.337493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:39.337728 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.337696 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:39.337728 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.337717 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:39.337728 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.337730 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zknj5 for pod openshift-network-diagnostics/network-check-target-289sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:39.337879 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.337792 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5 podName:3dbda061-df54-482d-96c6-851f1acf58a4 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:43.337773326 +0000 UTC m=+10.227420745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zknj5" (UniqueName: "kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5") pod "network-check-target-289sc" (UID: "3dbda061-df54-482d-96c6-851f1acf58a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:39.616656 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:39.616627 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:39.616802 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.616771 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:39.616875 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:39.616809 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:39.616944 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.616889 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:39.617012 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:39.616935 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:39.617065 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:39.617030 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:39.734599 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:39.733339 2582 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="77ac20d219726e226b4f910a446380641c5c8b7f86753c4fe4d9fc9ab58b8950" exitCode=0 Oct 29 21:41:39.734599 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:39.733496 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" event={"ID":"97611cb7-e267-4d11-b20a-c52d833cdefc","Type":"ContainerDied","Data":"77ac20d219726e226b4f910a446380641c5c8b7f86753c4fe4d9fc9ab58b8950"} Oct 29 21:41:39.817526 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:39.817103 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" podStartSLOduration=4.817079434 podStartE2EDuration="4.817079434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:38.770211437 +0000 UTC m=+5.659858867" watchObservedRunningTime="2026-10-29 21:41:39.817079434 +0000 UTC m=+6.706726862" Oct 29 21:41:40.767371 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:40.766579 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-krcw7" event={"ID":"73219cee-7208-4f68-ad0b-bc9cb0bcb199","Type":"ContainerStarted","Data":"8e2651edd0622665b5fad926306653c4913154fa8417056228bb51c1fe017e44"} Oct 29 21:41:40.813669 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:40.813424 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-krcw7" podStartSLOduration=3.054134321 podStartE2EDuration="6.813401129s" totalImagesPullingTime="3.759266808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:40.812526375 +0000 UTC m=+7.702173817" watchObservedRunningTime="2026-10-29 21:41:40.813401129 +0000 UTC m=+7.703048559" Oct 29 21:41:41.618191 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:41.618105 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:41.618191 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:41.618142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:41.618384 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:41.618294 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:41.618665 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:41.618493 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:41.618665 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:41.618625 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:41.618807 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:41.618710 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:41.771654 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:41.771621 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tjgwl" event={"ID":"89de847b-bcca-4f80-af6b-071681d3afdb","Type":"ContainerStarted","Data":"8cc36f5870c20878bb68a4753430fd478e5e7bfe8b9f4c3b48c8a68fc1eb6486"} Oct 29 21:41:42.775882 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:42.775842 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" event={"ID":"dcf1d737-3cce-4759-8bde-21c7e0dff0ae","Type":"ContainerStarted","Data":"2445739126cc8ff5ec3c09c6c221411b487fe2ba6febe441d7ad53ede539f373"} Oct 29 21:41:43.269587 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.269547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:43.269817 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.269603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:43.269817 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.269771 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:43.269948 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.269847 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret podName:fb5b0077-ddac-46d3-ba06-5fc065be5b53 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:51.269818881 +0000 UTC m=+18.159466300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret") pod "global-pull-secret-syncer-29g9k" (UID: "fb5b0077-ddac-46d3-ba06-5fc065be5b53") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:43.270571 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.270305 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:43.270571 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.270361 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:51.27034363 +0000 UTC m=+18.159991054 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:43.370164 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.370128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:43.370364 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.370346 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:43.370439 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.370373 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:43.370439 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.370387 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zknj5 for pod openshift-network-diagnostics/network-check-target-289sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:43.370551 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.370452 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5 podName:3dbda061-df54-482d-96c6-851f1acf58a4 nodeName:}" failed. No retries permitted until 2026-10-29 21:41:51.370432372 +0000 UTC m=+18.260079792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zknj5" (UniqueName: "kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5") pod "network-check-target-289sc" (UID: "3dbda061-df54-482d-96c6-851f1acf58a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:43.544665 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.544632 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 21:41:43.553280 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.553248 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:43.554160 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.554031 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:43.558960 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.558632 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:41:43.544659938Z","UUID":"ce20d63b-2054-4185-bff9-cb75cba790e3","Handler":null,"Name":"","Endpoint":""} Oct 29 21:41:43.561729 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.561285 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 21:41:43.561729 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.561317 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 21:41:43.604081 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.603993 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tjgwl" podStartSLOduration=4.48196593 podStartE2EDuration="9.603973639s" totalImagesPullingTime="5.122007709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:41.818594995 +0000 UTC m=+8.708242425" watchObservedRunningTime="2026-10-29 21:41:43.603973639 +0000 UTC m=+10.493621068" Oct 29 21:41:43.620021 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.619979 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:43.620185 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.620104 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:43.620472 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.620446 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:43.620549 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.620534 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:43.620602 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.620593 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:43.620697 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:43.620670 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:43.785688 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:43.785613 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" event={"ID":"dcf1d737-3cce-4759-8bde-21c7e0dff0ae","Type":"ContainerStarted","Data":"41f2b041178bf022ebf3997ac7960c228283548084e975ab7eefb99b08ce0c3d"} Oct 29 21:41:44.789760 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:44.789499 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s8t9b" event={"ID":"1a8cd583-ba4f-4f3a-9f42-f43d63462d32","Type":"ContainerStarted","Data":"b40711f05e4140ef53071ca5dbcb34cc1b3d5334a98071ef0c66c36f4cab7730"} Oct 29 21:41:44.795535 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:44.795009 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" event={"ID":"dcf1d737-3cce-4759-8bde-21c7e0dff0ae","Type":"ContainerStarted","Data":"2b6768745f92807154765ca8d5c3c007762d7a44abedee2ea8b3676efa600561"} Oct 29 21:41:44.797192 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:44.797161 2582 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="42e055b0bee43840468b0f5f0e72670637ff4b79f196e22a839874a9ae5b39be" exitCode=0 Oct 29 21:41:44.797294 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:44.797238 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" event={"ID":"97611cb7-e267-4d11-b20a-c52d833cdefc","Type":"ContainerDied","Data":"42e055b0bee43840468b0f5f0e72670637ff4b79f196e22a839874a9ae5b39be"} Oct 29 21:41:44.800789 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:44.800763 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t6k4p" event={"ID":"84058be2-867e-4c5a-a614-bccfd7af96fa","Type":"ContainerStarted","Data":"7872881362cdea87871f4f0e875bbdc4a9da51062b6d6ae40c99a2ec34506c6f"} Oct 29 21:41:44.885627 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:44.885571 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-s8t9b" podStartSLOduration=2.699391189 podStartE2EDuration="10.885552223s" totalImagesPullingTime="8.186161034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:44.835443795 +0000 UTC m=+11.725091219" watchObservedRunningTime="2026-10-29 21:41:44.885552223 +0000 UTC m=+11.775199648" Oct 29 21:41:44.927969 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:44.927860 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-t6k4p" podStartSLOduration=2.74915553 podStartE2EDuration="10.927845892s" totalImagesPullingTime="8.178690362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:44.9272362 +0000 UTC m=+11.816883629" watchObservedRunningTime="2026-10-29 21:41:44.927845892 +0000 UTC m=+11.817493317" Oct 29 21:41:45.618307 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:45.618269 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:45.618463 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:45.618386 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:45.618567 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:45.618462 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:45.618628 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:45.618579 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:45.618680 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:45.618644 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:45.618753 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:45.618729 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:45.804643 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:45.804603 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2s6z8" event={"ID":"4eb95095-a274-43e2-adbd-d88824033bb1","Type":"ContainerStarted","Data":"d83dae89078c5457d65a9451cc35c9438c9e6dd075f11eb00d1c5d1292582ffb"} Oct 29 21:41:45.807424 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:45.807397 2582 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="c4c40b580901952aadd901e06e917416155306049624533354264b1d95f6c85d" exitCode=0 Oct 29 21:41:45.807722 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:45.807693 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" event={"ID":"97611cb7-e267-4d11-b20a-c52d833cdefc","Type":"ContainerDied","Data":"c4c40b580901952aadd901e06e917416155306049624533354264b1d95f6c85d"} Oct 29 21:41:45.857425 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:45.857360 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" podStartSLOduration=3.539513838 podStartE2EDuration="11.85733755s" totalImagesPullingTime="8.317823712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:44.972671798 +0000 UTC m=+11.862319233" watchObservedRunningTime="2026-10-29 21:41:45.85733755 +0000 UTC m=+12.746984976" Oct 29 21:41:46.812672 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:46.812642 2582 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="4db962e1f813b9d8be9ef98cf91079d62a31918e68a930be2d9759bd36db492c" exitCode=0 Oct 29 21:41:46.813424 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:46.812722 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" event={"ID":"97611cb7-e267-4d11-b20a-c52d833cdefc","Type":"ContainerDied","Data":"4db962e1f813b9d8be9ef98cf91079d62a31918e68a930be2d9759bd36db492c"} Oct 29 21:41:46.818179 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:46.818137 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" event={"ID":"88711e24-d089-4fbb-9b36-fb015fe7f9c9","Type":"ContainerStarted","Data":"54a8f9e8db0a0227ea33b3ffe71f3858b8bcabac69a4b3a6428feef294d9f33a"} Oct 29 21:41:46.904728 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:46.904673 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" podStartSLOduration=2.387489094 podStartE2EDuration="12.904657747s" totalImagesPullingTime="10.517168653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:46.904170487 +0000 UTC m=+13.793817912" watchObservedRunningTime="2026-10-29 21:41:46.904657747 +0000 UTC m=+13.794305170" Oct 29 21:41:46.945503 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:46.945407 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2s6z8" podStartSLOduration=4.6722741580000005 podStartE2EDuration="12.945393483s" totalImagesPullingTime="8.273119325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:46.94476849 +0000 UTC m=+13.834415914" watchObservedRunningTime="2026-10-29 21:41:46.945393483 +0000 UTC m=+13.835040907" Oct 29 21:41:47.616250 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:47.616218 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:47.616250 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:47.616244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:47.616457 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:47.616339 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:47.616457 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:47.616395 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:47.616457 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:47.616446 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:47.616618 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:47.616555 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:49.616307 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:49.616072 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:49.616819 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:49.616084 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:49.616819 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:49.616386 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:49.616819 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:49.616129 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:49.616819 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:49.616467 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:49.616819 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:49.616566 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:49.833958 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:49.833920 2582 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="97cfd68c5243f5dbc339e7e335c45d4651d67f1be81610a693817df25579942a" exitCode=0 Oct 29 21:41:49.834129 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:49.833964 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" event={"ID":"97611cb7-e267-4d11-b20a-c52d833cdefc","Type":"ContainerDied","Data":"97cfd68c5243f5dbc339e7e335c45d4651d67f1be81610a693817df25579942a"} Oct 29 21:41:50.840318 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:50.840289 2582 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="dde8a0a0304bb3761b89d01f9e1919d7a273276876590433177c8b24e8708291" exitCode=0 Oct 29 21:41:50.840686 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:50.840324 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" event={"ID":"97611cb7-e267-4d11-b20a-c52d833cdefc","Type":"ContainerDied","Data":"dde8a0a0304bb3761b89d01f9e1919d7a273276876590433177c8b24e8708291"} Oct 29 21:41:51.332061 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.331881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:51.332171 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.332079 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:51.332171 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.332008 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:51.332171 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.332163 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:51.332302 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.332174 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.332159719 +0000 UTC m=+34.221807143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 21:41:51.332302 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.332198 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret podName:fb5b0077-ddac-46d3-ba06-5fc065be5b53 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.332188515 +0000 UTC m=+34.221835940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret") pod "global-pull-secret-syncer-29g9k" (UID: "fb5b0077-ddac-46d3-ba06-5fc065be5b53") : object "kube-system"/"original-pull-secret" not registered Oct 29 21:41:51.432731 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.432662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:51.432847 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.432821 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 21:41:51.432847 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.432842 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 21:41:51.432932 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.432852 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zknj5 for pod openshift-network-diagnostics/network-check-target-289sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:51.432932 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.432900 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5 podName:3dbda061-df54-482d-96c6-851f1acf58a4 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.432888059 +0000 UTC m=+34.322535460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zknj5" (UniqueName: "kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5") pod "network-check-target-289sc" (UID: "3dbda061-df54-482d-96c6-851f1acf58a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 21:41:51.616883 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.616852 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:51.616883 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.616890 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:51.617076 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.616893 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:51.617076 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.616998 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:51.617142 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.617123 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:51.617212 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:51.617195 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:51.845708 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.845676 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" event={"ID":"97611cb7-e267-4d11-b20a-c52d833cdefc","Type":"ContainerStarted","Data":"78f613fc6d0a34a437afd2b02ee308a0be128bec6316b8bd072ee5295613afa9"} Oct 29 21:41:51.848342 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.848320 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" event={"ID":"587e4eb6-4e54-4c6e-a640-71976f13827c","Type":"ContainerStarted","Data":"83fa5f6ac9c4ab629e4d9d8983edf294d8830d3d1cf13e9b4530404ac284d7f5"} Oct 29 21:41:51.848453 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.848378 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" event={"ID":"587e4eb6-4e54-4c6e-a640-71976f13827c","Type":"ContainerStarted","Data":"0b606629894b9d4cfa3cbf35fd697f3635271c04ca2bc6ea999bed3d4178b50d"} Oct 29 21:41:51.848453 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.848411 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" event={"ID":"587e4eb6-4e54-4c6e-a640-71976f13827c","Type":"ContainerStarted","Data":"33f3a6cc87163895e4718bc354534c9d6bc8d3568e2101c64c69c791f3fcb7e4"} Oct 29 21:41:51.848453 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.848439 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" event={"ID":"587e4eb6-4e54-4c6e-a640-71976f13827c","Type":"ContainerStarted","Data":"49581a8419d8588692b3b537aef3b95beec84013edd55024bcc11593446a8061"} Oct 29 21:41:51.848541 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.848470 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" event={"ID":"587e4eb6-4e54-4c6e-a640-71976f13827c","Type":"ContainerStarted","Data":"d6b7e8b59a00cb63012f0cc7da0d9df8f54e476f575c54dc580cf8963d84806b"} Oct 29 21:41:51.848541 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.848495 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" event={"ID":"587e4eb6-4e54-4c6e-a640-71976f13827c","Type":"ContainerStarted","Data":"ccd0ec8d1a0a6bd4d3dede1031c444270e1c52cd0c30d7a6d056b07e08059013"} Oct 29 21:41:51.894417 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:51.894365 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mdxzj" podStartSLOduration=7.038210368 podStartE2EDuration="17.894353203s" totalImagesPullingTime="10.856142835s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:51.894187905 +0000 UTC m=+18.783835329" watchObservedRunningTime="2026-10-29 21:41:51.894353203 +0000 UTC m=+18.784000627" Oct 29 21:41:53.616561 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:53.616479 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:53.617093 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:53.616572 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:53.617093 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:53.616654 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:53.617093 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:53.616741 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:53.617093 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:53.616787 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:53.617093 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:53.616854 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:53.855305 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:53.855265 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" event={"ID":"587e4eb6-4e54-4c6e-a640-71976f13827c","Type":"ContainerStarted","Data":"d666d54a9157c5ebd73ab44f088e49c679e1093b67914f80160a1f1cc36196e3"} Oct 29 21:41:55.616458 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:55.616428 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:55.617320 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:55.616468 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:55.617320 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:55.616475 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:55.617320 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:55.616632 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:55.617320 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:55.616717 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:55.617320 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:55.616764 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:55.862030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:55.861997 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" event={"ID":"587e4eb6-4e54-4c6e-a640-71976f13827c","Type":"ContainerStarted","Data":"6c47864fe2ff9db2155276d2cf30fdedcadf4259bb74bb93619eccaf6dccbdd5"} Oct 29 21:41:55.862310 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:55.862294 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:55.862371 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:55.862361 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:55.877168 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:55.877146 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:55.953804 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:55.953750 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" podStartSLOduration=6.857333811 podStartE2EDuration="21.953738571s" totalImagesPullingTime="15.09640476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:41:55.95365921 +0000 UTC m=+22.843306633" watchObservedRunningTime="2026-10-29 21:41:55.953738571 +0000 UTC m=+22.843385995" Oct 29 21:41:56.864534 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:56.864503 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:56.879007 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:56.878981 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:41:57.018670 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:57.018640 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:57.018829 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:57.018774 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 21:41:57.019171 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:57.019156 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:41:57.578449 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:57.578420 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-29g9k"] Oct 29 21:41:57.578668 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:57.578527 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:57.578668 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:57.578604 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:57.581250 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:57.581227 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-289sc"] Oct 29 21:41:57.581392 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:57.581326 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:57.581462 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:57.581397 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:57.584030 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:57.584010 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rcs9l"] Oct 29 21:41:57.584133 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:57.584096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:57.584196 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:57.584179 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:41:59.616797 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:59.616762 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:41:59.616797 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:59.616786 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:41:59.617340 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:41:59.616817 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:41:59.617340 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:59.616894 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:41:59.617340 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:59.616957 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:41:59.617340 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:41:59.616981 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:42:01.619403 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:01.619217 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:01.619826 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:01.619245 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:01.619826 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:01.619490 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:42:01.619826 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:01.619275 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:01.619826 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:01.619545 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:42:01.619826 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:01.619635 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:42:03.619855 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.619825 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:03.620293 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.619928 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:03.620293 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.619959 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:03.620293 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:03.619963 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29g9k" podUID="fb5b0077-ddac-46d3-ba06-5fc065be5b53" Oct 29 21:42:03.620293 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:03.620018 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289sc" podUID="3dbda061-df54-482d-96c6-851f1acf58a4" Oct 29 21:42:03.620293 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:03.620071 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcs9l" podUID="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" Oct 29 21:42:03.855016 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.854993 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeReady" Oct 29 21:42:03.855163 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.855113 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 21:42:03.906544 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.906515 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-42pjx"] Oct 29 21:42:03.909749 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.909724 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-42pjx" Oct 29 21:42:03.910163 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.910143 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mvw7m"] Oct 29 21:42:03.913311 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.913285 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b425c"] Oct 29 21:42:03.913435 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.913387 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mvw7m" Oct 29 21:42:03.916156 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.916137 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-859d6fc448-k9w4q"] Oct 29 21:42:03.916275 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.916254 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b425c" Oct 29 21:42:03.918882 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.918865 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-748b45bb85-pfdxr"] Oct 29 21:42:03.919001 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.918982 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:03.921687 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.921667 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zv2wb"] Oct 29 21:42:03.921787 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.921768 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:03.925656 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.925233 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mvwvt"] Oct 29 21:42:03.925656 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.925314 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:03.928322 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.928304 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56bd78965c-kscnl"] Oct 29 21:42:03.928441 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.928422 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:03.930847 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.930831 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 29 21:42:03.930938 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.930859 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 29 21:42:03.931777 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.931761 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6"] Oct 29 21:42:03.931915 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.931886 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:03.935602 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.935582 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64"] Oct 29 21:42:03.935731 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.935713 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:03.938537 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.938517 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6"] Oct 29 21:42:03.938655 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.938636 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:03.942370 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.942354 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk"] Oct 29 21:42:03.942481 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.942466 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:03.946330 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.946304 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5"] Oct 29 21:42:03.946582 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.946563 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk" Oct 29 21:42:03.950305 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.950287 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7jz5d"] Oct 29 21:42:03.950652 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.950624 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:03.951124 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.951105 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 29 21:42:03.951215 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.951138 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 29 21:42:03.951317 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.951241 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8g25n\"" Oct 29 21:42:03.951505 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.951346 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 29 21:42:03.954058 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.954038 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n"] Oct 29 21:42:03.954178 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.954162 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:03.957122 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.957106 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tfwmx"] Oct 29 21:42:03.957317 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.957239 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n" Oct 29 21:42:03.961719 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.961699 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mmxmn"] Oct 29 21:42:03.963926 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.962634 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:03.965222 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.965204 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-42pjx"] Oct 29 21:42:03.965372 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.965353 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mvw7m"] Oct 29 21:42:03.965466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.965383 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-859d6fc448-k9w4q"] Oct 29 21:42:03.965466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.965397 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mvwvt"] Oct 29 21:42:03.965466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.965409 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zv2wb"] Oct 29 21:42:03.965466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.965422 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n"] Oct 29 21:42:03.965466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.965362 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:03.966571 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.966547 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5"] Oct 29 21:42:03.970891 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.970871 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7jz5d"] Oct 29 21:42:03.971018 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.970935 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64"] Oct 29 21:42:03.972924 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.972885 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6"] Oct 29 21:42:03.997854 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:03.997828 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk"] Oct 29 21:42:04.004096 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.004073 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-748b45bb85-pfdxr"] Oct 29 21:42:04.009194 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.009166 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:04.009401 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.009384 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6"] Oct 29 21:42:04.009714 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.009690 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tfwmx"] Oct 29 21:42:04.012656 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.012619 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56bd78965c-kscnl"] Oct 29 21:42:04.018050 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.018029 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b425c"] Oct 29 21:42:04.021740 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.021719 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-config\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.021840 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.021751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887lk\" (UniqueName: \"kubernetes.io/projected/059a06d7-5255-47dc-8ce7-07d113e03ca1-kube-api-access-887lk\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.021840 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.021770 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj6l4\" (UniqueName: \"kubernetes.io/projected/61ac615e-a093-42ea-92a4-84d5c22c9f41-kube-api-access-zj6l4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:04.021981 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.021852 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.021981 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.021899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mngv9\" (UniqueName: \"kubernetes.io/projected/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-kube-api-access-mngv9\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:04.021981 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.021964 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-config\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:04.022129 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022094 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8ph\" (UniqueName: \"kubernetes.io/projected/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-kube-api-access-7x8ph\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:04.022235 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022200 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.022305 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.022364 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022319 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-default-certificate\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.022364 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022344 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91f12b29-bfab-4407-8cae-b618a658c23e-ca-trust-extracted\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.022474 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhssb\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-kube-api-access-qhssb\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.022474 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022441 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xdxh\" (UniqueName: \"kubernetes.io/projected/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-kube-api-access-7xdxh\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.022621 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-trusted-ca\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.022621 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022559 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-tmp\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.022621 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61ac615e-a093-42ea-92a4-84d5c22c9f41-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:04.022771 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-image-registry-private-configuration\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.022771 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022680 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-bound-sa-token\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.022771 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022708 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lzfl\" (UniqueName: \"kubernetes.io/projected/33291806-5f11-4f13-ba59-92a143fe8f0e-kube-api-access-5lzfl\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:04.022771 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022743 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.022978 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.022978 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022797 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-installation-pull-secrets\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.022978 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022829 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsvhn\" (UniqueName: \"kubernetes.io/projected/3398371c-c328-4876-963c-8af098bae440-kube-api-access-rsvhn\") pod \"kas-connection-checker-685895c7c9-42pjx\" (UID: \"3398371c-c328-4876-963c-8af098bae440\") " pod="kube-system/kas-connection-checker-685895c7c9-42pjx" Oct 29 21:42:04.022978 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-registry-certificates\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.022978 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022865 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrjk\" (UniqueName: \"kubernetes.io/projected/20070f80-4b65-40ff-9646-e8b7fef85cc1-kube-api-access-xgrjk\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.022978 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022891 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e889f05-b993-4423-9b9c-4673ad6e70d3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:04.023346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.022976 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61ac615e-a093-42ea-92a4-84d5c22c9f41-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:04.023346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-serving-cert\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:04.023346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023073 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4k74\" (UniqueName: \"kubernetes.io/projected/f8539056-ee01-4262-89b4-1d4b55a5073c-kube-api-access-x4k74\") pod \"volume-data-source-validator-58c9f5f46d-jtzgk\" (UID: \"f8539056-ee01-4262-89b4-1d4b55a5073c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk" Oct 29 21:42:04.023346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-serving-cert\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.023346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:04.023346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023188 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-bound-sa-token\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.023346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-certificates\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.023346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20070f80-4b65-40ff-9646-e8b7fef85cc1-serving-cert\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.023346 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023322 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-stats-auth\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.023792 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023381 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-installation-pull-secrets\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.023792 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023430 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kflk\" (UniqueName: \"kubernetes.io/projected/3dd58b31-fa23-4a8c-b113-ad9479a7b681-kube-api-access-2kflk\") pod \"kas-connection-checker-685895c7c9-b425c\" (UID: \"3dd58b31-fa23-4a8c-b113-ad9479a7b681\") " pod="kube-system/kas-connection-checker-685895c7c9-b425c" Oct 29 21:42:04.023792 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023536 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:04.023792 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:04.023792 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023673 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-image-registry-private-configuration\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.023792 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.023792 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023772 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv7n9\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-kube-api-access-wv7n9\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.023792 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023794 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-service-ca-bundle\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.024398 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023816 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:04.024398 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-trusted-ca\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.024398 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-snapshots\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.024398 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.023979 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9h8r\" (UniqueName: \"kubernetes.io/projected/f04d5d43-128a-4972-b55e-836175e60130-kube-api-access-m9h8r\") pod \"kas-connection-checker-685895c7c9-mvw7m\" (UID: \"f04d5d43-128a-4972-b55e-836175e60130\") " pod="kube-system/kas-connection-checker-685895c7c9-mvw7m" Oct 29 21:42:04.024398 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.024027 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-ca-trust-extracted\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.024398 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.024053 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/33291806-5f11-4f13-ba59-92a143fe8f0e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:04.030172 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.030151 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 29 21:42:04.030257 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.030161 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:04.030324 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.030297 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:04.030371 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.030345 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 29 21:42:04.030773 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.030756 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 21:42:04.031577 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.031548 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 29 21:42:04.031685 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.031503 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 21:42:04.031917 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.031883 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:04.031999 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.031976 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 21:42:04.032064 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.032047 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 29 21:42:04.032113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.032096 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:04.032368 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.032353 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:04.032949 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.032514 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:04.032949 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.032534 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:04.032949 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.032616 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:04.032949 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.032693 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 21:42:04.032949 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.032832 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 29 21:42:04.033647 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.032984 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 21:42:04.033647 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.033088 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 21:42:04.033647 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.033137 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 21:42:04.033647 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.033308 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 21:42:04.033647 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.033310 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:04.033647 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.033416 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:04.033647 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.033505 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 29 21:42:04.033647 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.033590 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 29 21:42:04.033998 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.033717 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 21:42:04.033998 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.033746 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 21:42:04.033998 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.033784 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 21:42:04.045019 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.045002 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 29 21:42:04.057200 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057177 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 29 21:42:04.057345 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057179 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 29 21:42:04.057345 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057244 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 29 21:42:04.057345 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057269 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 29 21:42:04.057345 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057288 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 29 21:42:04.057345 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057297 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 29 21:42:04.057345 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057290 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 29 21:42:04.057345 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057326 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 29 21:42:04.057345 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057330 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 29 21:42:04.057345 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057335 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 29 21:42:04.057671 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057350 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 29 21:42:04.057671 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057371 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 29 21:42:04.057671 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057383 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 29 21:42:04.057671 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057420 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 21:42:04.057671 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057547 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bt4z6\"" Oct 29 21:42:04.057671 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057610 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r7b2w\"" Oct 29 21:42:04.057852 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057782 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vqm2\"" Oct 29 21:42:04.057852 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.057829 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 21:42:04.058361 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.058345 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mmxmn"] Oct 29 21:42:04.125296 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-default-certificate\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.125296 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125245 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91f12b29-bfab-4407-8cae-b618a658c23e-ca-trust-extracted\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.125296 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhssb\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-kube-api-access-qhssb\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.125296 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xdxh\" (UniqueName: \"kubernetes.io/projected/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-kube-api-access-7xdxh\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.125627 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-trusted-ca\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.125627 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-tmp\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.125627 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125369 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61ac615e-a093-42ea-92a4-84d5c22c9f41-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:04.125627 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-image-registry-private-configuration\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.125627 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-bound-sa-token\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.125857 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125702 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91f12b29-bfab-4407-8cae-b618a658c23e-ca-trust-extracted\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.125857 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lzfl\" (UniqueName: \"kubernetes.io/projected/33291806-5f11-4f13-ba59-92a143fe8f0e-kube-api-access-5lzfl\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:04.125857 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125765 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgk9\" (UniqueName: \"kubernetes.io/projected/b39d65b2-cca6-4f08-bdec-becd6e000c48-kube-api-access-kcgk9\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.125857 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125796 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qnfp\" (UniqueName: \"kubernetes.io/projected/1461042e-6499-452c-b548-cb243c31b17d-kube-api-access-6qnfp\") pod \"ingress-canary-tfwmx\" (UID: \"1461042e-6499-452c-b548-cb243c31b17d\") " pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:04.125857 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125828 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.125857 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125838 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-tmp\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.125857 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125854 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1461042e-6499-452c-b548-cb243c31b17d-cert\") pod \"ingress-canary-tfwmx\" (UID: \"1461042e-6499-452c-b548-cb243c31b17d\") " pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:04.126190 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.125932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.126190 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.125966 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:04.126190 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.125978 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748b45bb85-pfdxr: secret "image-registry-tls" not found Oct 29 21:42:04.126190 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126003 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-installation-pull-secrets\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.126190 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsvhn\" (UniqueName: \"kubernetes.io/projected/3398371c-c328-4876-963c-8af098bae440-kube-api-access-rsvhn\") pod \"kas-connection-checker-685895c7c9-42pjx\" (UID: \"3398371c-c328-4876-963c-8af098bae440\") " pod="kube-system/kas-connection-checker-685895c7c9-42pjx" Oct 29 21:42:04.126190 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-registry-certificates\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.126190 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.126115 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls podName:5ca158db-9c67-4fb5-bc6b-7c15ceee8772 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:04.626068623 +0000 UTC m=+31.515716042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls") pod "image-registry-748b45bb85-pfdxr" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772") : secret "image-registry-tls" not found Oct 29 21:42:04.126190 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.126193 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:04.126503 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.126205 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859d6fc448-k9w4q: secret "image-registry-tls" not found Oct 29 21:42:04.126503 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.126267 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls podName:91f12b29-bfab-4407-8cae-b618a658c23e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:04.626250589 +0000 UTC m=+31.515898008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls") pod "image-registry-859d6fc448-k9w4q" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e") : secret "image-registry-tls" not found Oct 29 21:42:04.126503 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126302 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrjk\" (UniqueName: \"kubernetes.io/projected/20070f80-4b65-40ff-9646-e8b7fef85cc1-kube-api-access-xgrjk\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.126503 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e889f05-b993-4423-9b9c-4673ad6e70d3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:04.126503 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126360 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61ac615e-a093-42ea-92a4-84d5c22c9f41-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:04.126503 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-serving-cert\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:04.126503 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126395 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-trusted-ca\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.126503 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.126503 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4k74\" (UniqueName: \"kubernetes.io/projected/f8539056-ee01-4262-89b4-1d4b55a5073c-kube-api-access-x4k74\") pod \"volume-data-source-validator-58c9f5f46d-jtzgk\" (UID: \"f8539056-ee01-4262-89b4-1d4b55a5073c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk" Oct 29 21:42:04.126503 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126473 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-serving-cert\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126504 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-bound-sa-token\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-certificates\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20070f80-4b65-40ff-9646-e8b7fef85cc1-serving-cert\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126621 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzv72\" (UniqueName: \"kubernetes.io/projected/1b583a61-cc48-4c4e-9e2e-7f6e684d45e8-kube-api-access-qzv72\") pod \"network-check-source-6f78d59f99-zc52n\" (UID: \"1b583a61-cc48-4c4e-9e2e-7f6e684d45e8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126645 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-stats-auth\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-installation-pull-secrets\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126700 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-registry-certificates\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kflk\" (UniqueName: \"kubernetes.io/projected/3dd58b31-fa23-4a8c-b113-ad9479a7b681-kube-api-access-2kflk\") pod \"kas-connection-checker-685895c7c9-b425c\" (UID: \"3dd58b31-fa23-4a8c-b113-ad9479a7b681\") " pod="kube-system/kas-connection-checker-685895c7c9-b425c" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126751 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.126763 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126779 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39d65b2-cca6-4f08-bdec-becd6e000c48-tmp-dir\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.126820 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert podName:1e889f05-b993-4423-9b9c-4673ad6e70d3 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:04.626802337 +0000 UTC m=+31.516449760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zv2wb" (UID: "1e889f05-b993-4423-9b9c-4673ad6e70d3") : secret "networking-console-plugin-cert" not found Oct 29 21:42:04.126873 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.126844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-image-registry-private-configuration\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv7n9\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-kube-api-access-wv7n9\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-service-ca-bundle\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127157 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61ac615e-a093-42ea-92a4-84d5c22c9f41-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127181 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-trusted-ca\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127247 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-snapshots\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39d65b2-cca6-4f08-bdec-becd6e000c48-config-volume\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127334 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9h8r\" (UniqueName: \"kubernetes.io/projected/f04d5d43-128a-4972-b55e-836175e60130-kube-api-access-m9h8r\") pod \"kas-connection-checker-685895c7c9-mvw7m\" (UID: \"f04d5d43-128a-4972-b55e-836175e60130\") " pod="kube-system/kas-connection-checker-685895c7c9-mvw7m" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127405 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-ca-trust-extracted\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/33291806-5f11-4f13-ba59-92a143fe8f0e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:04.127487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127461 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e889f05-b993-4423-9b9c-4673ad6e70d3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:04.128113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-config\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.128113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887lk\" (UniqueName: \"kubernetes.io/projected/059a06d7-5255-47dc-8ce7-07d113e03ca1-kube-api-access-887lk\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.128113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj6l4\" (UniqueName: \"kubernetes.io/projected/61ac615e-a093-42ea-92a4-84d5c22c9f41-kube-api-access-zj6l4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:04.128113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127721 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.128113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mngv9\" (UniqueName: \"kubernetes.io/projected/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-kube-api-access-mngv9\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:04.128113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-config\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:04.128113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127839 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8ph\" (UniqueName: \"kubernetes.io/projected/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-kube-api-access-7x8ph\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:04.128113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127867 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.128113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.127958 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:04.128113 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.128008 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.128586 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.128270 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/33291806-5f11-4f13-ba59-92a143fe8f0e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:04.128586 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.128298 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:04.628281191 +0000 UTC m=+31.517928616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : configmap references non-existent config key: service-ca.crt Oct 29 21:42:04.130615 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.129823 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-config\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:04.131580 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.130781 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.131580 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.130876 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.131580 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.131142 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61ac615e-a093-42ea-92a4-84d5c22c9f41-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:04.131580 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.131205 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-certificates\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.131580 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.131476 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20070f80-4b65-40ff-9646-e8b7fef85cc1-serving-cert\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.131580 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.131501 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 21:42:04.131580 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.131551 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:04.631534822 +0000 UTC m=+31.521182244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : secret "router-metrics-certs-default" not found Oct 29 21:42:04.132068 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.131816 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-trusted-ca\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.132068 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.131825 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-installation-pull-secrets\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.132068 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.131856 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-default-certificate\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.132068 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.131930 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:04.132068 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.131937 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-stats-auth\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.132068 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.131945 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 21:42:04.132068 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.131974 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls podName:33291806-5f11-4f13-ba59-92a143fe8f0e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:04.631960225 +0000 UTC m=+31.521607638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-m4ks6" (UID: "33291806-5f11-4f13-ba59-92a143fe8f0e") : secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:04.132068 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.131995 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls podName:53ff731b-5c02-4366-b6b2-c5ecfdc8be38 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:04.63198358 +0000 UTC m=+31.521630996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-46z64" (UID: "53ff731b-5c02-4366-b6b2-c5ecfdc8be38") : secret "samples-operator-tls" not found Oct 29 21:42:04.132445 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.132299 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-ca-trust-extracted\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.132445 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.132317 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-service-ca-bundle\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.132672 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.132632 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-snapshots\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.132775 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.132676 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-config\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.132958 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.132937 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-serving-cert\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:04.133115 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.133064 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-image-registry-private-configuration\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.133115 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.133066 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-image-registry-private-configuration\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.134084 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.134061 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-installation-pull-secrets\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.134344 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.134326 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-serving-cert\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.152950 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.152923 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsvhn\" (UniqueName: \"kubernetes.io/projected/3398371c-c328-4876-963c-8af098bae440-kube-api-access-rsvhn\") pod \"kas-connection-checker-685895c7c9-42pjx\" (UID: \"3398371c-c328-4876-963c-8af098bae440\") " pod="kube-system/kas-connection-checker-685895c7c9-42pjx" Oct 29 21:42:04.153134 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.153012 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrjk\" (UniqueName: \"kubernetes.io/projected/20070f80-4b65-40ff-9646-e8b7fef85cc1-kube-api-access-xgrjk\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.153450 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.153429 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xdxh\" (UniqueName: \"kubernetes.io/projected/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-kube-api-access-7xdxh\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.153581 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.153560 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhssb\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-kube-api-access-qhssb\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.155124 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.155103 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngv9\" (UniqueName: \"kubernetes.io/projected/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-kube-api-access-mngv9\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:04.155344 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.155326 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj6l4\" (UniqueName: \"kubernetes.io/projected/61ac615e-a093-42ea-92a4-84d5c22c9f41-kube-api-access-zj6l4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:04.155408 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.155347 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887lk\" (UniqueName: \"kubernetes.io/projected/059a06d7-5255-47dc-8ce7-07d113e03ca1-kube-api-access-887lk\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.156796 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.156769 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8ph\" (UniqueName: \"kubernetes.io/projected/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-kube-api-access-7x8ph\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:04.158182 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.158136 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4k74\" (UniqueName: \"kubernetes.io/projected/f8539056-ee01-4262-89b4-1d4b55a5073c-kube-api-access-x4k74\") pod \"volume-data-source-validator-58c9f5f46d-jtzgk\" (UID: \"f8539056-ee01-4262-89b4-1d4b55a5073c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk" Oct 29 21:42:04.158334 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.158284 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9h8r\" (UniqueName: \"kubernetes.io/projected/f04d5d43-128a-4972-b55e-836175e60130-kube-api-access-m9h8r\") pod \"kas-connection-checker-685895c7c9-mvw7m\" (UID: \"f04d5d43-128a-4972-b55e-836175e60130\") " pod="kube-system/kas-connection-checker-685895c7c9-mvw7m" Oct 29 21:42:04.158402 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.158379 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-bound-sa-token\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.158402 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.158392 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lzfl\" (UniqueName: \"kubernetes.io/projected/33291806-5f11-4f13-ba59-92a143fe8f0e-kube-api-access-5lzfl\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:04.160097 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.160072 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kflk\" (UniqueName: \"kubernetes.io/projected/3dd58b31-fa23-4a8c-b113-ad9479a7b681-kube-api-access-2kflk\") pod \"kas-connection-checker-685895c7c9-b425c\" (UID: \"3dd58b31-fa23-4a8c-b113-ad9479a7b681\") " pod="kube-system/kas-connection-checker-685895c7c9-b425c" Oct 29 21:42:04.160509 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.160492 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv7n9\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-kube-api-access-wv7n9\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.162504 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.162486 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-bound-sa-token\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.223296 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.223268 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-42pjx" Oct 29 21:42:04.228702 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.228671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39d65b2-cca6-4f08-bdec-becd6e000c48-config-volume\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.228977 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.228824 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgk9\" (UniqueName: \"kubernetes.io/projected/b39d65b2-cca6-4f08-bdec-becd6e000c48-kube-api-access-kcgk9\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.228977 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.228849 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qnfp\" (UniqueName: \"kubernetes.io/projected/1461042e-6499-452c-b548-cb243c31b17d-kube-api-access-6qnfp\") pod \"ingress-canary-tfwmx\" (UID: \"1461042e-6499-452c-b548-cb243c31b17d\") " pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:04.228977 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.228884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1461042e-6499-452c-b548-cb243c31b17d-cert\") pod \"ingress-canary-tfwmx\" (UID: \"1461042e-6499-452c-b548-cb243c31b17d\") " pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:04.228977 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.228945 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.229179 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.228985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzv72\" (UniqueName: \"kubernetes.io/projected/1b583a61-cc48-4c4e-9e2e-7f6e684d45e8-kube-api-access-qzv72\") pod \"network-check-source-6f78d59f99-zc52n\" (UID: \"1b583a61-cc48-4c4e-9e2e-7f6e684d45e8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n" Oct 29 21:42:04.229179 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.229021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39d65b2-cca6-4f08-bdec-becd6e000c48-tmp-dir\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.229179 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.229069 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:04.229179 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.229131 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls podName:b39d65b2-cca6-4f08-bdec-becd6e000c48 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:04.729112888 +0000 UTC m=+31.618760305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls") pod "dns-default-mmxmn" (UID: "b39d65b2-cca6-4f08-bdec-becd6e000c48") : secret "dns-default-metrics-tls" not found Oct 29 21:42:04.229371 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.229332 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39d65b2-cca6-4f08-bdec-becd6e000c48-config-volume\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.229413 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.229390 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39d65b2-cca6-4f08-bdec-becd6e000c48-tmp-dir\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.231088 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.231069 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mvw7m" Oct 29 21:42:04.231397 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.231380 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1461042e-6499-452c-b548-cb243c31b17d-cert\") pod \"ingress-canary-tfwmx\" (UID: \"1461042e-6499-452c-b548-cb243c31b17d\") " pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:04.249990 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.249762 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b425c" Oct 29 21:42:04.253207 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.253066 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:04.255848 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.255653 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzv72\" (UniqueName: \"kubernetes.io/projected/1b583a61-cc48-4c4e-9e2e-7f6e684d45e8-kube-api-access-qzv72\") pod \"network-check-source-6f78d59f99-zc52n\" (UID: \"1b583a61-cc48-4c4e-9e2e-7f6e684d45e8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n" Oct 29 21:42:04.260756 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.260731 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qnfp\" (UniqueName: \"kubernetes.io/projected/1461042e-6499-452c-b548-cb243c31b17d-kube-api-access-6qnfp\") pod \"ingress-canary-tfwmx\" (UID: \"1461042e-6499-452c-b548-cb243c31b17d\") " pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:04.262599 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.262581 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgk9\" (UniqueName: \"kubernetes.io/projected/b39d65b2-cca6-4f08-bdec-becd6e000c48-kube-api-access-kcgk9\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.276272 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.276014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:04.300047 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.299783 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:04.311709 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.311484 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk" Oct 29 21:42:04.318108 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.318039 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:04.324541 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.324421 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:04.330797 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.330705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n" Oct 29 21:42:04.339397 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.339194 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:04.557820 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.557709 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mvw7m"] Oct 29 21:42:04.558700 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.558672 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-42pjx"] Oct 29 21:42:04.559780 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:04.559754 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3398371c_c328_4876_963c_8af098bae440.slice/crio-1643973be0e38523f3b9cd7c13f4c66a87c0a4f3af1abb9493379cacd093a2b5 WatchSource:0}: Error finding container 1643973be0e38523f3b9cd7c13f4c66a87c0a4f3af1abb9493379cacd093a2b5: Status 404 returned error can't find the container with id 1643973be0e38523f3b9cd7c13f4c66a87c0a4f3af1abb9493379cacd093a2b5 Oct 29 21:42:04.560342 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:04.560325 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04d5d43_128a_4972_b55e_836175e60130.slice/crio-c47d9ccf1cd78d617c085e1aac96743be4175a29223f17b2ac31a3de366b2d06 WatchSource:0}: Error finding container c47d9ccf1cd78d617c085e1aac96743be4175a29223f17b2ac31a3de366b2d06: Status 404 returned error can't find the container with id c47d9ccf1cd78d617c085e1aac96743be4175a29223f17b2ac31a3de366b2d06 Oct 29 21:42:04.604506 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.604267 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mvwvt"] Oct 29 21:42:04.614601 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:04.614569 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4b9c11_19cd_46b9_8096_5c9375fa7e91.slice/crio-cff6966fdb10b6799cd5ef1268fe57b84f35ce5f0d450ee04156ff27af404ca3 WatchSource:0}: Error finding container cff6966fdb10b6799cd5ef1268fe57b84f35ce5f0d450ee04156ff27af404ca3: Status 404 returned error can't find the container with id cff6966fdb10b6799cd5ef1268fe57b84f35ce5f0d450ee04156ff27af404ca3 Oct 29 21:42:04.621532 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.619261 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b425c"] Oct 29 21:42:04.624679 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:04.624101 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd58b31_fa23_4a8c_b113_ad9479a7b681.slice/crio-bd32bbcea0e44cba9598bff7d80e116317a22a9369a9baccdf142a05d35b5211 WatchSource:0}: Error finding container bd32bbcea0e44cba9598bff7d80e116317a22a9369a9baccdf142a05d35b5211: Status 404 returned error can't find the container with id bd32bbcea0e44cba9598bff7d80e116317a22a9369a9baccdf142a05d35b5211 Oct 29 21:42:04.625440 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.625395 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6"] Oct 29 21:42:04.627647 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:04.627599 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc5b397_30ee_49b3_b52a_00e1fa2b6cf9.slice/crio-c4050a5c505d4eb0544fc2048219393ba857834c1918d2af92f4d522f779e95e WatchSource:0}: Error finding container c4050a5c505d4eb0544fc2048219393ba857834c1918d2af92f4d522f779e95e: Status 404 returned error can't find the container with id c4050a5c505d4eb0544fc2048219393ba857834c1918d2af92f4d522f779e95e Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.634133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.634182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.634214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.634239 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.634324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.634364 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.634390 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.634551 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.634566 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859d6fc448-k9w4q: secret "image-registry-tls" not found Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.634619 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls podName:91f12b29-bfab-4407-8cae-b618a658c23e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:05.634599287 +0000 UTC m=+32.524246692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls") pod "image-registry-859d6fc448-k9w4q" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e") : secret "image-registry-tls" not found Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635335 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635382 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert podName:1e889f05-b993-4423-9b9c-4673ad6e70d3 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:05.635367156 +0000 UTC m=+32.525014575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zv2wb" (UID: "1e889f05-b993-4423-9b9c-4673ad6e70d3") : secret "networking-console-plugin-cert" not found Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635437 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635471 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls podName:33291806-5f11-4f13-ba59-92a143fe8f0e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:05.635460332 +0000 UTC m=+32.525107750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-m4ks6" (UID: "33291806-5f11-4f13-ba59-92a143fe8f0e") : secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:04.636135 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635521 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 21:42:04.637001 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635550 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:05.635540494 +0000 UTC m=+32.525187899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : secret "router-metrics-certs-default" not found Oct 29 21:42:04.637001 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635608 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 21:42:04.637001 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635636 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls podName:53ff731b-5c02-4366-b6b2-c5ecfdc8be38 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:05.635627048 +0000 UTC m=+32.525274452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-46z64" (UID: "53ff731b-5c02-4366-b6b2-c5ecfdc8be38") : secret "samples-operator-tls" not found Oct 29 21:42:04.637001 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635692 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:05.635682853 +0000 UTC m=+32.525330254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : configmap references non-existent config key: service-ca.crt Oct 29 21:42:04.637001 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635748 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:04.637001 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635759 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748b45bb85-pfdxr: secret "image-registry-tls" not found Oct 29 21:42:04.637001 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.635787 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls podName:5ca158db-9c67-4fb5-bc6b-7c15ceee8772 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:05.635777894 +0000 UTC m=+32.525425297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls") pod "image-registry-748b45bb85-pfdxr" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772") : secret "image-registry-tls" not found Oct 29 21:42:04.698438 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.698407 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7jz5d"] Oct 29 21:42:04.704393 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:04.704357 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20070f80_4b65_40ff_9646_e8b7fef85cc1.slice/crio-4497a77913293bef0400eae4eea98a88726ab3a897a4669a8c046458f8fe4a0c WatchSource:0}: Error finding container 4497a77913293bef0400eae4eea98a88726ab3a897a4669a8c046458f8fe4a0c: Status 404 returned error can't find the container with id 4497a77913293bef0400eae4eea98a88726ab3a897a4669a8c046458f8fe4a0c Oct 29 21:42:04.709463 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.709435 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk"] Oct 29 21:42:04.719350 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:04.719325 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8539056_ee01_4262_89b4_1d4b55a5073c.slice/crio-bf5bf574f821b4f7acabc9a2da7906642ebc55e0677f7310a51b36ecdb56a350 WatchSource:0}: Error finding container bf5bf574f821b4f7acabc9a2da7906642ebc55e0677f7310a51b36ecdb56a350: Status 404 returned error can't find the container with id bf5bf574f821b4f7acabc9a2da7906642ebc55e0677f7310a51b36ecdb56a350 Oct 29 21:42:04.720137 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.719974 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tfwmx"] Oct 29 21:42:04.724523 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:04.724162 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1461042e_6499_452c_b548_cb243c31b17d.slice/crio-dc552fd029be282c995addbb015fc26b60f5e32ad52197dc897acbc1048c9473 WatchSource:0}: Error finding container dc552fd029be282c995addbb015fc26b60f5e32ad52197dc897acbc1048c9473: Status 404 returned error can't find the container with id dc552fd029be282c995addbb015fc26b60f5e32ad52197dc897acbc1048c9473 Oct 29 21:42:04.731236 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.731218 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n"] Oct 29 21:42:04.736652 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.733730 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5"] Oct 29 21:42:04.736652 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:04.733737 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b583a61_cc48_4c4e_9e2e_7f6e684d45e8.slice/crio-21a7875621d9fd575976eb9219c5bcb0bc9c53f3a294f76732386708a875a1e3 WatchSource:0}: Error finding container 21a7875621d9fd575976eb9219c5bcb0bc9c53f3a294f76732386708a875a1e3: Status 404 returned error can't find the container with id 21a7875621d9fd575976eb9219c5bcb0bc9c53f3a294f76732386708a875a1e3 Oct 29 21:42:04.736652 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.735050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:04.736652 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.735188 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:04.736652 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:04.735251 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls podName:b39d65b2-cca6-4f08-bdec-becd6e000c48 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:05.735232798 +0000 UTC m=+32.624880203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls") pod "dns-default-mmxmn" (UID: "b39d65b2-cca6-4f08-bdec-becd6e000c48") : secret "dns-default-metrics-tls" not found Oct 29 21:42:04.737143 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:04.737119 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ac615e_a093_42ea_92a4_84d5c22c9f41.slice/crio-1b3c6dd04afb96255e3827b95c08c4ffe5059c6b36948b812223e700619d59fa WatchSource:0}: Error finding container 1b3c6dd04afb96255e3827b95c08c4ffe5059c6b36948b812223e700619d59fa: Status 404 returned error can't find the container with id 1b3c6dd04afb96255e3827b95c08c4ffe5059c6b36948b812223e700619d59fa Oct 29 21:42:04.881343 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.881309 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n" event={"ID":"1b583a61-cc48-4c4e-9e2e-7f6e684d45e8","Type":"ContainerStarted","Data":"21a7875621d9fd575976eb9219c5bcb0bc9c53f3a294f76732386708a875a1e3"} Oct 29 21:42:04.882469 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.882447 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk" event={"ID":"f8539056-ee01-4262-89b4-1d4b55a5073c","Type":"ContainerStarted","Data":"bf5bf574f821b4f7acabc9a2da7906642ebc55e0677f7310a51b36ecdb56a350"} Oct 29 21:42:04.883721 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.883655 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tfwmx" event={"ID":"1461042e-6499-452c-b548-cb243c31b17d","Type":"ContainerStarted","Data":"dc552fd029be282c995addbb015fc26b60f5e32ad52197dc897acbc1048c9473"} Oct 29 21:42:04.884776 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.884757 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" event={"ID":"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9","Type":"ContainerStarted","Data":"c4050a5c505d4eb0544fc2048219393ba857834c1918d2af92f4d522f779e95e"} Oct 29 21:42:04.885833 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.885814 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mvwvt" event={"ID":"1f4b9c11-19cd-46b9-8096-5c9375fa7e91","Type":"ContainerStarted","Data":"cff6966fdb10b6799cd5ef1268fe57b84f35ce5f0d450ee04156ff27af404ca3"} Oct 29 21:42:04.887147 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.887128 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b425c" event={"ID":"3dd58b31-fa23-4a8c-b113-ad9479a7b681","Type":"ContainerStarted","Data":"0ff0a5c7468c4be22d78101a7da28fe196c1fcc525e5a38b0b0b7142b86efd96"} Oct 29 21:42:04.887214 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.887164 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b425c" event={"ID":"3dd58b31-fa23-4a8c-b113-ad9479a7b681","Type":"ContainerStarted","Data":"bd32bbcea0e44cba9598bff7d80e116317a22a9369a9baccdf142a05d35b5211"} Oct 29 21:42:04.889960 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.889938 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mvw7m" event={"ID":"f04d5d43-128a-4972-b55e-836175e60130","Type":"ContainerStarted","Data":"f49d46f3285c472889a41cc1493ad1930542e25cf76de8d51f17fcdc2692c09c"} Oct 29 21:42:04.890045 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.889981 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mvw7m" event={"ID":"f04d5d43-128a-4972-b55e-836175e60130","Type":"ContainerStarted","Data":"c47d9ccf1cd78d617c085e1aac96743be4175a29223f17b2ac31a3de366b2d06"} Oct 29 21:42:04.891407 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.891388 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-42pjx" event={"ID":"3398371c-c328-4876-963c-8af098bae440","Type":"ContainerStarted","Data":"b32eb59928b48da4112e8bdada9fc080a1f8a7d523636d7be9b1d3d4a36079e5"} Oct 29 21:42:04.891487 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.891428 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-42pjx" event={"ID":"3398371c-c328-4876-963c-8af098bae440","Type":"ContainerStarted","Data":"1643973be0e38523f3b9cd7c13f4c66a87c0a4f3af1abb9493379cacd093a2b5"} Oct 29 21:42:04.892478 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.892456 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" event={"ID":"20070f80-4b65-40ff-9646-e8b7fef85cc1","Type":"ContainerStarted","Data":"4497a77913293bef0400eae4eea98a88726ab3a897a4669a8c046458f8fe4a0c"} Oct 29 21:42:04.897364 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.897337 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" event={"ID":"61ac615e-a093-42ea-92a4-84d5c22c9f41","Type":"ContainerStarted","Data":"1b3c6dd04afb96255e3827b95c08c4ffe5059c6b36948b812223e700619d59fa"} Oct 29 21:42:04.930304 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.930247 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-b425c" podStartSLOduration=694.930234231 podStartE2EDuration="11m34.930234231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:04.929094472 +0000 UTC m=+31.818741919" watchObservedRunningTime="2026-10-29 21:42:04.930234231 +0000 UTC m=+31.819881652" Oct 29 21:42:04.971761 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:04.971706 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-42pjx" podStartSLOduration=694.971680698 podStartE2EDuration="11m34.971680698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:04.971471428 +0000 UTC m=+31.861118866" watchObservedRunningTime="2026-10-29 21:42:04.971680698 +0000 UTC m=+31.861328121" Oct 29 21:42:05.014737 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.014671 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mvw7m" podStartSLOduration=695.014651224 podStartE2EDuration="11m35.014651224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:05.013304411 +0000 UTC m=+31.902951835" watchObservedRunningTime="2026-10-29 21:42:05.014651224 +0000 UTC m=+31.904298650" Oct 29 21:42:05.557334 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.557297 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:05.617082 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.617047 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:05.617997 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.617724 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:05.619931 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.618577 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:05.634488 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.634430 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 21:42:05.635556 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.635080 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lrv4h\"" Oct 29 21:42:05.635556 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.635236 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 21:42:05.635556 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.635432 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gdpmh\"" Oct 29 21:42:05.635944 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.635892 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cqgbm\"" Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.651704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.651765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.651797 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.651846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.651892 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.651942 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.651971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652129 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652183 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls podName:53ff731b-5c02-4366-b6b2-c5ecfdc8be38 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.65216741 +0000 UTC m=+34.541814815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-46z64" (UID: "53ff731b-5c02-4366-b6b2-c5ecfdc8be38") : secret "samples-operator-tls" not found Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652259 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.652249453 +0000 UTC m=+34.541896858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : configmap references non-existent config key: service-ca.crt Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652308 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652318 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748b45bb85-pfdxr: secret "image-registry-tls" not found Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652348 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls podName:5ca158db-9c67-4fb5-bc6b-7c15ceee8772 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.652338195 +0000 UTC m=+34.541985597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls") pod "image-registry-748b45bb85-pfdxr" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772") : secret "image-registry-tls" not found Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652427 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652437 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859d6fc448-k9w4q: secret "image-registry-tls" not found Oct 29 21:42:05.652958 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652467 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls podName:91f12b29-bfab-4407-8cae-b618a658c23e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.652457469 +0000 UTC m=+34.542104872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls") pod "image-registry-859d6fc448-k9w4q" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e") : secret "image-registry-tls" not found Oct 29 21:42:05.653682 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652526 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 21:42:05.653682 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652558 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert podName:1e889f05-b993-4423-9b9c-4673ad6e70d3 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.652547564 +0000 UTC m=+34.542194969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zv2wb" (UID: "1e889f05-b993-4423-9b9c-4673ad6e70d3") : secret "networking-console-plugin-cert" not found Oct 29 21:42:05.653682 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652609 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:05.653682 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652638 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls podName:33291806-5f11-4f13-ba59-92a143fe8f0e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.652628611 +0000 UTC m=+34.542276016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-m4ks6" (UID: "33291806-5f11-4f13-ba59-92a143fe8f0e") : secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:05.653682 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652687 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 21:42:05.653682 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.652715 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.652706239 +0000 UTC m=+34.542353643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : secret "router-metrics-certs-default" not found Oct 29 21:42:05.741199 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.741113 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:05.753224 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:05.752868 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:05.753224 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.753126 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:05.753224 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:05.753188 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls podName:b39d65b2-cca6-4f08-bdec-becd6e000c48 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:07.753169506 +0000 UTC m=+34.642816921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls") pod "dns-default-mmxmn" (UID: "b39d65b2-cca6-4f08-bdec-becd6e000c48") : secret "dns-default-metrics-tls" not found Oct 29 21:42:06.315433 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:06.315356 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:06.976300 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:06.976231 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:07.154546 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.154517 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:07.373497 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.373451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:07.373685 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.373516 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:07.374010 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.373987 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 21:42:07.374127 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.374110 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:39.374093431 +0000 UTC m=+66.263740836 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : secret "metrics-daemon-secret" not found Oct 29 21:42:07.376960 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.376935 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:07.474368 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.474327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:07.478690 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.478665 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:07.495875 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.495839 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:07.676466 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.676424 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:07.676621 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.676480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:07.676621 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.676558 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:07.676621 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.676601 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:07.676779 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.676628 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:07.676779 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.676670 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:07.676779 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.676711 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:07.676937 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.676816 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:07.676937 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.676874 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls podName:33291806-5f11-4f13-ba59-92a143fe8f0e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.676859034 +0000 UTC m=+38.566506458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-m4ks6" (UID: "33291806-5f11-4f13-ba59-92a143fe8f0e") : secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:07.677280 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677262 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 21:42:07.677361 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677304 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.677293159 +0000 UTC m=+38.566940560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : secret "router-metrics-certs-default" not found Oct 29 21:42:07.677361 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677341 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 21:42:07.677361 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677360 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls podName:53ff731b-5c02-4366-b6b2-c5ecfdc8be38 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.677354448 +0000 UTC m=+38.567001850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-46z64" (UID: "53ff731b-5c02-4366-b6b2-c5ecfdc8be38") : secret "samples-operator-tls" not found Oct 29 21:42:07.677511 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677400 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.677394308 +0000 UTC m=+38.567041710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : configmap references non-existent config key: service-ca.crt Oct 29 21:42:07.677511 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677444 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:07.677511 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677456 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748b45bb85-pfdxr: secret "image-registry-tls" not found Oct 29 21:42:07.677511 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677480 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls podName:5ca158db-9c67-4fb5-bc6b-7c15ceee8772 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.677474136 +0000 UTC m=+38.567121541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls") pod "image-registry-748b45bb85-pfdxr" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772") : secret "image-registry-tls" not found Oct 29 21:42:07.677691 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677518 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:07.677691 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677524 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859d6fc448-k9w4q: secret "image-registry-tls" not found Oct 29 21:42:07.677691 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677540 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls podName:91f12b29-bfab-4407-8cae-b618a658c23e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.677535708 +0000 UTC m=+38.567183110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls") pod "image-registry-859d6fc448-k9w4q" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e") : secret "image-registry-tls" not found Oct 29 21:42:07.677691 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677572 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 21:42:07.677691 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.677590 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert podName:1e889f05-b993-4423-9b9c-4673ad6e70d3 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.677585138 +0000 UTC m=+38.567232539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zv2wb" (UID: "1e889f05-b993-4423-9b9c-4673ad6e70d3") : secret "networking-console-plugin-cert" not found Oct 29 21:42:07.680679 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.680651 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:07.759974 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.759645 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:07.778284 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.778254 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:07.778462 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.778409 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:07.778530 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:07.778478 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls podName:b39d65b2-cca6-4f08-bdec-becd6e000c48 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.7784604 +0000 UTC m=+38.668107807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls") pod "dns-default-mmxmn" (UID: "b39d65b2-cca6-4f08-bdec-becd6e000c48") : secret "dns-default-metrics-tls" not found Oct 29 21:42:07.874721 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.874497 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-29g9k"] Oct 29 21:42:07.915326 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.914484 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk" event={"ID":"f8539056-ee01-4262-89b4-1d4b55a5073c","Type":"ContainerStarted","Data":"1bbf81d7cefbddda835f966eb37827e0061c6f37eca79fa2b1e1ef77ece0de99"} Oct 29 21:42:07.918139 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.918077 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-29g9k" event={"ID":"fb5b0077-ddac-46d3-ba06-5fc065be5b53","Type":"ContainerStarted","Data":"d1db0e66a80973c5eaba40d1897880bbd7e3e5a886b4ec23d97fc3cdd76d6b5c"} Oct 29 21:42:07.958051 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:07.957992 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk" podStartSLOduration=666.747619877 podStartE2EDuration="11m8.95797063s" totalImagesPullingTime="2.210350753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:07.957614633 +0000 UTC m=+34.847262058" watchObservedRunningTime="2026-10-29 21:42:07.95797063 +0000 UTC m=+34.847618052" Oct 29 21:42:08.103174 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:08.103143 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-289sc"] Oct 29 21:42:08.106167 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:08.106131 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbda061_df54_482d_96c6_851f1acf58a4.slice/crio-e349b10ee8d9a8abebdd54178777d129826acf750219dec77bee8286311d500e WatchSource:0}: Error finding container e349b10ee8d9a8abebdd54178777d129826acf750219dec77bee8286311d500e: Status 404 returned error can't find the container with id e349b10ee8d9a8abebdd54178777d129826acf750219dec77bee8286311d500e Oct 29 21:42:08.139857 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:08.139756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:08.313943 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:08.313878 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:08.925370 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:08.925332 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-289sc" event={"ID":"3dbda061-df54-482d-96c6-851f1acf58a4","Type":"ContainerStarted","Data":"e349b10ee8d9a8abebdd54178777d129826acf750219dec77bee8286311d500e"} Oct 29 21:42:09.935498 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:09.934668 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" event={"ID":"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9","Type":"ContainerStarted","Data":"7a09fea05bfd19caa84d02e47d66805bbef77fd0dbe720ba91f00ecb2275b675"} Oct 29 21:42:09.938758 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:09.938724 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mvwvt" event={"ID":"1f4b9c11-19cd-46b9-8096-5c9375fa7e91","Type":"ContainerStarted","Data":"6a657c3b0fd89cd5726b0d6c81aa459ceb79fe911b016815216432cf1ec1e827"} Oct 29 21:42:09.941503 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:09.941479 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/0.log" Oct 29 21:42:09.941635 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:09.941524 2582 generic.go:356] "Generic (PLEG): container finished" podID="20070f80-4b65-40ff-9646-e8b7fef85cc1" containerID="1fd0fadeed80c8ab327e060c0b1171960c88b1de5dcea8b8dc67a474c7411157" exitCode=255 Oct 29 21:42:09.941635 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:09.941570 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" event={"ID":"20070f80-4b65-40ff-9646-e8b7fef85cc1","Type":"ContainerDied","Data":"1fd0fadeed80c8ab327e060c0b1171960c88b1de5dcea8b8dc67a474c7411157"} Oct 29 21:42:09.941799 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:09.941780 2582 scope.go:122] "RemoveContainer" containerID="1fd0fadeed80c8ab327e060c0b1171960c88b1de5dcea8b8dc67a474c7411157" Oct 29 21:42:09.984404 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:09.984342 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" podStartSLOduration=666.015936259 podStartE2EDuration="11m10.984322185s" totalImagesPullingTime="4.968385926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:09.983209033 +0000 UTC m=+36.872856458" watchObservedRunningTime="2026-10-29 21:42:09.984322185 +0000 UTC m=+36.873969613" Oct 29 21:42:10.029523 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.029466 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-mvwvt" podStartSLOduration=666.105599886 podStartE2EDuration="11m11.029449465s" totalImagesPullingTime="4.923849579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:10.028750575 +0000 UTC m=+36.918398002" watchObservedRunningTime="2026-10-29 21:42:10.029449465 +0000 UTC m=+36.919096895" Oct 29 21:42:10.872844 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.872767 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8dsqf"] Oct 29 21:42:10.896924 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.896438 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:10.898182 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.898109 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8dsqf"] Oct 29 21:42:10.926496 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.926410 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 21:42:10.926654 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.926578 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lk29w\"" Oct 29 21:42:10.926764 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.926738 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:10.926938 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.926801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg6zw\" (UniqueName: \"kubernetes.io/projected/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-api-access-qg6zw\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:10.927014 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.926947 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:10.927069 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.927026 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4dbf16f-ae81-461b-8564-99fce7202e3c-data-volume\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:10.927123 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.927107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4dbf16f-ae81-461b-8564-99fce7202e3c-crio-socket\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:10.951888 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.951854 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tfwmx" event={"ID":"1461042e-6499-452c-b548-cb243c31b17d","Type":"ContainerStarted","Data":"8a4135a6d0a3501b8381674da09db71a9756c9a166f3db71f470b00b47262046"} Oct 29 21:42:10.954532 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.954498 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/1.log" Oct 29 21:42:10.955080 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.955057 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/0.log" Oct 29 21:42:10.955197 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.955102 2582 generic.go:356] "Generic (PLEG): container finished" podID="20070f80-4b65-40ff-9646-e8b7fef85cc1" containerID="b66b5d7d234cfd8206a074cdeb2ee9e4cc101586f990e55edee853f3b12fd063" exitCode=255 Oct 29 21:42:10.955835 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.955666 2582 scope.go:122] "RemoveContainer" containerID="b66b5d7d234cfd8206a074cdeb2ee9e4cc101586f990e55edee853f3b12fd063" Oct 29 21:42:10.955835 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.955684 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" event={"ID":"20070f80-4b65-40ff-9646-e8b7fef85cc1","Type":"ContainerDied","Data":"b66b5d7d234cfd8206a074cdeb2ee9e4cc101586f990e55edee853f3b12fd063"} Oct 29 21:42:10.955835 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.955759 2582 scope.go:122] "RemoveContainer" containerID="1fd0fadeed80c8ab327e060c0b1171960c88b1de5dcea8b8dc67a474c7411157" Oct 29 21:42:10.956064 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:10.955880 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7jz5d_openshift-console-operator(20070f80-4b65-40ff-9646-e8b7fef85cc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" podUID="20070f80-4b65-40ff-9646-e8b7fef85cc1" Oct 29 21:42:10.959972 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.959945 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" event={"ID":"61ac615e-a093-42ea-92a4-84d5c22c9f41","Type":"ContainerStarted","Data":"4d8df66619ac1dff88fac39bc8625093e69181aa87e5ffc74d97a433c4aeb749"} Oct 29 21:42:10.996201 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.996149 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tfwmx" podStartSLOduration=2.542960223 podStartE2EDuration="7.996133953s" totalImagesPullingTime="5.45317373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:10.995963967 +0000 UTC m=+37.885611393" watchObservedRunningTime="2026-10-29 21:42:10.996133953 +0000 UTC m=+37.885781381" Oct 29 21:42:10.997104 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:10.997081 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 21:42:11.012758 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.012725 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:11.030598 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.030201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:11.030598 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.030259 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg6zw\" (UniqueName: \"kubernetes.io/projected/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-api-access-qg6zw\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:11.030598 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.030330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:11.030598 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.030396 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4dbf16f-ae81-461b-8564-99fce7202e3c-data-volume\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:11.030598 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.030569 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 21:42:11.030598 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.030577 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4dbf16f-ae81-461b-8564-99fce7202e3c-crio-socket\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:11.030974 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.030618 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm podName:a4dbf16f-ae81-461b-8564-99fce7202e3c nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.530602491 +0000 UTC m=+38.420249899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8dsqf" (UID: "a4dbf16f-ae81-461b-8564-99fce7202e3c") : configmap "kube-rbac-proxy" not found Oct 29 21:42:11.030974 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.030819 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:11.030974 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.030874 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls podName:a4dbf16f-ae81-461b-8564-99fce7202e3c nodeName:}" failed. No retries permitted until 2026-10-29 21:42:11.530856765 +0000 UTC m=+38.420504182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8dsqf" (UID: "a4dbf16f-ae81-461b-8564-99fce7202e3c") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:11.032012 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.031956 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4dbf16f-ae81-461b-8564-99fce7202e3c-crio-socket\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:11.032012 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.031993 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4dbf16f-ae81-461b-8564-99fce7202e3c-data-volume\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:11.048394 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.048352 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" podStartSLOduration=666.630308943 podStartE2EDuration="11m12.048339316s" totalImagesPullingTime="5.418030373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:11.046878297 +0000 UTC m=+37.936525722" watchObservedRunningTime="2026-10-29 21:42:11.048339316 +0000 UTC m=+37.937986795" Oct 29 21:42:11.057172 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.057122 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg6zw\" (UniqueName: \"kubernetes.io/projected/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-api-access-qg6zw\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:11.212259 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.212185 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:11.390869 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.390828 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:11.538968 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.538873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:11.539119 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.539016 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 21:42:11.539119 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.539085 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:11.539119 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.539103 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm podName:a4dbf16f-ae81-461b-8564-99fce7202e3c nodeName:}" failed. No retries permitted until 2026-10-29 21:42:12.539084261 +0000 UTC m=+39.428731678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8dsqf" (UID: "a4dbf16f-ae81-461b-8564-99fce7202e3c") : configmap "kube-rbac-proxy" not found Oct 29 21:42:11.539285 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.539197 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:11.539285 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.539248 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls podName:a4dbf16f-ae81-461b-8564-99fce7202e3c nodeName:}" failed. No retries permitted until 2026-10-29 21:42:12.539233552 +0000 UTC m=+39.428880957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8dsqf" (UID: "a4dbf16f-ae81-461b-8564-99fce7202e3c") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:11.642533 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.642496 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:11.659662 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.659637 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4"] Oct 29 21:42:11.684123 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.683888 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4"] Oct 29 21:42:11.684240 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.684088 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4" Oct 29 21:42:11.741519 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.741486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:11.741519 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.741524 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bql\" (UniqueName: \"kubernetes.io/projected/ec6efbe6-a797-4508-a997-c1b3aceec331-kube-api-access-v9bql\") pod \"migrator-66df5d496b-b65z4\" (UID: \"ec6efbe6-a797-4508-a997-c1b3aceec331\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4" Oct 29 21:42:11.741741 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.741614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:11.741741 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.741666 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 21:42:11.741741 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.741701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:11.741741 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.741731 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls podName:53ff731b-5c02-4366-b6b2-c5ecfdc8be38 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.741710353 +0000 UTC m=+46.631357768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-46z64" (UID: "53ff731b-5c02-4366-b6b2-c5ecfdc8be38") : secret "samples-operator-tls" not found Oct 29 21:42:11.741977 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.741769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:11.741977 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.741787 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:11.741977 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.741800 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748b45bb85-pfdxr: secret "image-registry-tls" not found Oct 29 21:42:11.741977 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.741812 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.741794684 +0000 UTC m=+46.631442102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : configmap references non-existent config key: service-ca.crt Oct 29 21:42:11.741977 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.741835 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls podName:5ca158db-9c67-4fb5-bc6b-7c15ceee8772 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.7418263 +0000 UTC m=+46.631473701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls") pod "image-registry-748b45bb85-pfdxr" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772") : secret "image-registry-tls" not found Oct 29 21:42:11.741977 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.741865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:11.741977 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.741935 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:11.741977 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.741872 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:11.741977 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.741961 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 21:42:11.742345 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.742018 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 21:42:11.742345 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.741966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:11.742345 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.742029 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:11.742345 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.742021 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert podName:1e889f05-b993-4423-9b9c-4673ad6e70d3 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.742006316 +0000 UTC m=+46.631653734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zv2wb" (UID: "1e889f05-b993-4423-9b9c-4673ad6e70d3") : secret "networking-console-plugin-cert" not found Oct 29 21:42:11.742345 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.741966 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859d6fc448-k9w4q: secret "image-registry-tls" not found Oct 29 21:42:11.742345 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.742121 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.742103585 +0000 UTC m=+46.631750992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : secret "router-metrics-certs-default" not found Oct 29 21:42:11.742345 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.742140 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls podName:33291806-5f11-4f13-ba59-92a143fe8f0e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.742130417 +0000 UTC m=+46.631777824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-m4ks6" (UID: "33291806-5f11-4f13-ba59-92a143fe8f0e") : secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:11.742345 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.742156 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls podName:91f12b29-bfab-4407-8cae-b618a658c23e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.742147824 +0000 UTC m=+46.631795233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls") pod "image-registry-859d6fc448-k9w4q" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e") : secret "image-registry-tls" not found Oct 29 21:42:11.788823 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.788794 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 29 21:42:11.788964 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.788855 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:11.789125 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.789084 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jxdl2\"" Oct 29 21:42:11.825804 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.825771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:11.842977 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.842950 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:11.843109 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.843048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bql\" (UniqueName: \"kubernetes.io/projected/ec6efbe6-a797-4508-a997-c1b3aceec331-kube-api-access-v9bql\") pod \"migrator-66df5d496b-b65z4\" (UID: \"ec6efbe6-a797-4508-a997-c1b3aceec331\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4" Oct 29 21:42:11.843175 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.843125 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:11.843223 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.843196 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls podName:b39d65b2-cca6-4f08-bdec-becd6e000c48 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.843177813 +0000 UTC m=+46.732825231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls") pod "dns-default-mmxmn" (UID: "b39d65b2-cca6-4f08-bdec-becd6e000c48") : secret "dns-default-metrics-tls" not found Oct 29 21:42:11.870654 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.870594 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bql\" (UniqueName: \"kubernetes.io/projected/ec6efbe6-a797-4508-a997-c1b3aceec331-kube-api-access-v9bql\") pod \"migrator-66df5d496b-b65z4\" (UID: \"ec6efbe6-a797-4508-a997-c1b3aceec331\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4" Oct 29 21:42:11.968213 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.968189 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/1.log" Oct 29 21:42:11.968622 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.968543 2582 scope.go:122] "RemoveContainer" containerID="b66b5d7d234cfd8206a074cdeb2ee9e4cc101586f990e55edee853f3b12fd063" Oct 29 21:42:11.968738 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:11.968716 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7jz5d_openshift-console-operator(20070f80-4b65-40ff-9646-e8b7fef85cc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" podUID="20070f80-4b65-40ff-9646-e8b7fef85cc1" Oct 29 21:42:11.970143 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.970107 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n" event={"ID":"1b583a61-cc48-4c4e-9e2e-7f6e684d45e8","Type":"ContainerStarted","Data":"98fc52be10249a3fd6f7d280679b0cfc27311946700888bca75ca9b57d280eee"} Oct 29 21:42:11.971982 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.971934 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-289sc" event={"ID":"3dbda061-df54-482d-96c6-851f1acf58a4","Type":"ContainerStarted","Data":"fe424eda076145570ef0757462e382011a735749fb381d263f6e580e879c0a70"} Oct 29 21:42:11.972292 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:11.972273 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:12.008994 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:12.008972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4" Oct 29 21:42:12.015972 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:12.015943 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:12.060735 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:12.060630 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-289sc" podStartSLOduration=34.556894128 podStartE2EDuration="38.060610786s" totalImagesPullingTime="3.503716658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:12.058061084 +0000 UTC m=+38.947708515" watchObservedRunningTime="2026-10-29 21:42:12.060610786 +0000 UTC m=+38.950258266" Oct 29 21:42:12.212142 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:12.212106 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:12.330610 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:12.330552 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n" podStartSLOduration=666.839645165 podStartE2EDuration="11m13.33053143s" totalImagesPullingTime="6.490886265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:12.102632694 +0000 UTC m=+38.992280162" watchObservedRunningTime="2026-10-29 21:42:12.33053143 +0000 UTC m=+39.220178854" Oct 29 21:42:12.331924 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:12.331884 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4"] Oct 29 21:42:12.334342 ip-10-0-130-134 kubenswrapper[2582]: W1029 21:42:12.334310 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6efbe6_a797_4508_a997_c1b3aceec331.slice/crio-71ee871b237a77d68af20d3963de811db9c0123270ca5ba9aaa071d830bef998 WatchSource:0}: Error finding container 71ee871b237a77d68af20d3963de811db9c0123270ca5ba9aaa071d830bef998: Status 404 returned error can't find the container with id 71ee871b237a77d68af20d3963de811db9c0123270ca5ba9aaa071d830bef998 Oct 29 21:42:12.550818 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:12.550783 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:12.551022 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:12.550848 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:12.551022 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:12.550961 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 21:42:12.551137 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:12.551048 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm podName:a4dbf16f-ae81-461b-8564-99fce7202e3c nodeName:}" failed. No retries permitted until 2026-10-29 21:42:14.551029242 +0000 UTC m=+41.440676657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8dsqf" (UID: "a4dbf16f-ae81-461b-8564-99fce7202e3c") : configmap "kube-rbac-proxy" not found Oct 29 21:42:12.551137 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:12.550966 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:12.551137 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:12.551101 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls podName:a4dbf16f-ae81-461b-8564-99fce7202e3c nodeName:}" failed. No retries permitted until 2026-10-29 21:42:14.551089246 +0000 UTC m=+41.440736658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8dsqf" (UID: "a4dbf16f-ae81-461b-8564-99fce7202e3c") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:12.845767 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:12.845731 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:12.977549 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:12.977511 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4" event={"ID":"ec6efbe6-a797-4508-a997-c1b3aceec331","Type":"ContainerStarted","Data":"71ee871b237a77d68af20d3963de811db9c0123270ca5ba9aaa071d830bef998"} Oct 29 21:42:13.982621 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:13.982587 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4" event={"ID":"ec6efbe6-a797-4508-a997-c1b3aceec331","Type":"ContainerStarted","Data":"00aa089225c77e31142eeb8b91437a887d0969b5cb4730b06709d480ce48e544"} Oct 29 21:42:14.025282 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:14.025240 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4" podStartSLOduration=1.542568277 podStartE2EDuration="3.025227173s" totalImagesPullingTime="1.482658896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:14.02405045 +0000 UTC m=+40.913697871" watchObservedRunningTime="2026-10-29 21:42:14.025227173 +0000 UTC m=+40.914874596" Oct 29 21:42:14.325499 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:14.325467 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:14.325646 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:14.325526 2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:14.325843 ip-10-0-130-134 kubenswrapper[2582]: I1029 21:42:14.325831 2582 scope.go:122] "RemoveContainer" containerID="b66b5d7d234cfd8206a074cdeb2ee9e4cc101586f990e55edee853f3b12fd063" Oct 29 21:42:14.326066 ip-10-0-130-134 kubenswrapper[2582]: E1029 21:42:14.326036 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7jz5d_openshift-console-operator(20070f80-4b65-40ff-9646-e8b7fef85cc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" podUID="20070f80-4b65-40ff-9646-e8b7fef85cc1" Oct 29 21:42:14.458456 ip-10-0-130-134 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 21:42:14.476232 ip-10-0-130-134 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 21:42:14.476391 ip-10-0-130-134 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 21:42:14.476436 ip-10-0-130-134 systemd[1]: kubelet.service: Consumed 3.059s CPU time. Oct 29 21:42:14.495508 ip-10-0-130-134 systemd[1]: Starting Kubernetes Kubelet... Oct 29 21:42:14.565195 ip-10-0-130-134 kubenswrapper[4952]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:42:14.565195 ip-10-0-130-134 kubenswrapper[4952]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 21:42:14.565195 ip-10-0-130-134 kubenswrapper[4952]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:42:14.565195 ip-10-0-130-134 kubenswrapper[4952]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 21:42:14.567239 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567163 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:14.567239 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567239 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:14.567239 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567243 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567246 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567249 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567252 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567255 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567258 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567261 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567263 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567266 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567269 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567271 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567274 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567276 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567279 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567281 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567286 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567289 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567292 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567295 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567298 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:14.567341 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567306 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567309 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567312 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567314 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567317 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567319 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567322 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567324 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567327 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567330 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567332 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567335 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567338 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567340 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567345 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567349 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567352 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567356 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567358 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:14.568479 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567361 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567364 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567366 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567369 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567371 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567374 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567377 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567379 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567381 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567384 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567387 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567389 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567392 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567395 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567403 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567406 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567409 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567412 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567415 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:14.568970 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567417 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567420 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567422 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567425 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567428 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567430 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567432 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567435 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567437 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567440 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567442 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567445 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567447 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567450 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567452 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567455 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567457 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567460 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567463 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567466 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:14.569426 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567468 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567471 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567473 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567476 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567478 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567484 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567486 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567489 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567497 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567500 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567503 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567506 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567509 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567511 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567514 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567517 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567519 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567522 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567524 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567527 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:14.569913 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567530 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567532 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567535 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567537 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567539 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.567542 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568374 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568383 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568386 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568389 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568392 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568394 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568397 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568400 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568402 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568405 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568407 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568412 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568415 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568418 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:14.570407 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568422 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568431 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568433 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568436 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568439 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568441 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568444 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568460 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568465 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568468 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568471 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568474 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568477 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568480 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568482 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568485 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568488 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568491 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568494 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:14.570958 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568496 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568499 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568502 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568504 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568507 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568509 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568512 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568516 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568521 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568524 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568528 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568531 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568534 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568537 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568540 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568549 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568552 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568554 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568557 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:14.571425 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568559 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568562 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568564 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568567 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568570 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568572 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568576 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568579 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568583 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568587 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568590 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568593 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568596 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568599 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568602 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568611 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568614 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568616 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568618 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:14.571899 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568621 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568624 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568626 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568629 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568632 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568635 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568638 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568641 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568644 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568646 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568656 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568659 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568661 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568664 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568666 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568669 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568671 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568674 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568677 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568680 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:14.572404 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568682 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568684 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568687 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568689 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568692 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568694 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568697 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568699 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568702 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568705 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568707 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568710 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568712 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568716 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.568718 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569295 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569303 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569307 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569310 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:14.572879 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569313 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569315 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569318 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569321 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569324 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569326 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569329 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569332 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569335 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569337 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569340 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569343 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569345 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569348 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569350 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569353 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569355 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569358 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569360 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:14.573392 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569363 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569365 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569368 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569370 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569373 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569375 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569378 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569380 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569382 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569385 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569389 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569392 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569394 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569397 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569400 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569402 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569405 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569407 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569410 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569413 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:14.573862 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569415 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569429 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569433 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569436 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569439 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569442 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569445 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569448 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569450 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569453 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569455 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569460 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569462 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569465 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569467 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569470 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569472 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569475 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569477 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569480 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:14.574359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569483 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569485 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569490 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569493 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569495 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569498 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569500 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569503 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569506 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569509 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569517 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569520 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569522 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569525 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569528 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569530 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569532 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569535 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569537 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569540 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:14.574839 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569542 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569545 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569548 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569552 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569555 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569557 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569560 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569562 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569565 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569569 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569572 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569575 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569585 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569588 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569592 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569595 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569597 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569600 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569602 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:14.575359 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569605 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569609 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569613 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.569616 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569712 4952 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569719 4952 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569729 4952 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569733 4952 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569737 4952 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569740 4952 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569745 4952 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569749 4952 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569752 4952 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569755 4952 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569758 4952 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569762 4952 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569766 4952 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569769 4952 flags.go:64] FLAG: --cgroup-root="" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569772 4952 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569775 4952 flags.go:64] FLAG: --client-ca-file="" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569778 4952 flags.go:64] FLAG: --cloud-provider="external" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569781 4952 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569788 4952 flags.go:64] FLAG: --cluster-domain="" Oct 29 21:42:14.576138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569791 4952 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569794 4952 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569811 4952 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569815 4952 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569822 4952 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569825 4952 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569828 4952 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569832 4952 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569835 4952 flags.go:64] FLAG: --contention-profiling="false" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569838 4952 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569840 4952 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569844 4952 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569846 4952 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569851 4952 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569859 4952 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569862 4952 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569865 4952 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569868 4952 flags.go:64] FLAG: --enable-server="true" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569871 4952 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569879 4952 flags.go:64] FLAG: --event-burst="100" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569882 4952 flags.go:64] FLAG: --event-qps="50" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569885 4952 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569888 4952 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569891 4952 flags.go:64] FLAG: --eviction-hard="" Oct 29 21:42:14.577051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569896 4952 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569912 4952 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569916 4952 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569919 4952 flags.go:64] FLAG: --eviction-soft="" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569923 4952 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569926 4952 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569929 4952 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569932 4952 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569935 4952 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569938 4952 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569941 4952 flags.go:64] FLAG: --feature-gates="" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569945 4952 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569949 4952 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569952 4952 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569955 4952 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569958 4952 flags.go:64] FLAG: --healthz-port="10248" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569961 4952 flags.go:64] FLAG: --help="false" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569964 4952 flags.go:64] FLAG: --hostname-override="ip-10-0-130-134.ec2.internal" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569967 4952 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569970 4952 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569973 4952 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569976 4952 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569980 4952 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569988 4952 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 21:42:14.577673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569991 4952 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569994 4952 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.569997 4952 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570001 4952 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570004 4952 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570006 4952 flags.go:64] FLAG: --kube-reserved="" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570009 4952 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570012 4952 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570015 4952 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570019 4952 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570022 4952 flags.go:64] FLAG: --lock-file="" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570025 4952 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570028 4952 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570031 4952 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570036 4952 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570039 4952 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570041 4952 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570044 4952 flags.go:64] FLAG: --logging-format="text" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570047 4952 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570051 4952 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570054 4952 flags.go:64] FLAG: --manifest-url="" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570057 4952 flags.go:64] FLAG: --manifest-url-header="" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570066 4952 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570069 4952 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570073 4952 flags.go:64] FLAG: --max-pods="110" Oct 29 21:42:14.578301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570076 4952 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570079 4952 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570081 4952 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570084 4952 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570087 4952 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570090 4952 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570093 4952 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570105 4952 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570109 4952 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570112 4952 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570115 4952 flags.go:64] FLAG: --pod-cidr="" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570118 4952 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570120 4952 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570124 4952 flags.go:64] FLAG: --pods-per-core="0" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570127 4952 flags.go:64] FLAG: --port="10250" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570130 4952 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570134 4952 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc7dd9c89c62e7d7" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570137 4952 flags.go:64] FLAG: --qos-reserved="" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570140 4952 flags.go:64] FLAG: --read-only-port="10255" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570143 4952 flags.go:64] FLAG: --register-node="true" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570146 4952 flags.go:64] FLAG: --register-with-taints="" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570150 4952 flags.go:64] FLAG: --registry-burst="10" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570153 4952 flags.go:64] FLAG: --registry-qps="5" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570156 4952 flags.go:64] FLAG: --reserved-cpus="" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570159 4952 flags.go:64] FLAG: --reserved-memory="" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570163 4952 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 21:42:14.578970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570166 4952 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570169 4952 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570172 4952 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570175 4952 flags.go:64] FLAG: --runonce="false" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570178 4952 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570181 4952 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570184 4952 flags.go:64] FLAG: --seccomp-default="false" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570187 4952 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570190 4952 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570193 4952 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570196 4952 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570199 4952 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570202 4952 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570205 4952 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570215 4952 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570218 4952 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570221 4952 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570224 4952 flags.go:64] FLAG: --system-cgroups="" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570227 4952 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570232 4952 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570235 4952 flags.go:64] FLAG: --tls-cert-file="" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570238 4952 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570246 4952 flags.go:64] FLAG: --tls-min-version="" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570249 4952 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570252 4952 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 21:42:14.579590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570254 4952 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570257 4952 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570260 4952 flags.go:64] FLAG: --v="2" Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570264 4952 flags.go:64] FLAG: --version="false" Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570268 4952 flags.go:64] FLAG: --vmodule="" Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570272 4952 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570276 4952 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570471 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570475 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570478 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570481 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570484 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570487 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570490 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570492 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570495 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570498 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570500 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570503 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570505 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570508 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:14.580417 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570510 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570519 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570521 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570524 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570528 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570531 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570534 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570536 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570541 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570544 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570547 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570549 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570552 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570554 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570558 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570562 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570565 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570569 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570572 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:14.581048 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570575 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570578 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570581 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570583 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570586 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570588 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570591 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570593 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570596 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570598 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570601 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570604 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570606 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570609 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570611 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570619 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570622 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570626 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570628 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:14.581582 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570631 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570634 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570638 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570641 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570643 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570646 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570649 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570651 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570654 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570657 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570659 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570662 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570664 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570667 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570670 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570672 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570675 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570677 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570680 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570683 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:14.582138 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570685 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570688 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570690 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570693 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570696 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570698 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570701 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570703 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570706 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570715 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570720 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570722 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570725 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570728 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570733 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570735 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570738 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570741 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570743 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:14.582673 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570749 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570752 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570755 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570757 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570760 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570762 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570765 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570767 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570770 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570772 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570775 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570777 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570780 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570782 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570785 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.570810 4952 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:42:14.583331 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570855 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570859 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570862 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570864 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570867 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570869 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570872 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570880 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570884 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570887 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570890 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570912 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570915 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570918 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570921 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570923 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570926 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570929 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570932 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570934 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:14.583786 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570937 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570940 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570942 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570945 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570948 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570950 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570953 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570956 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570958 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570961 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570970 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570973 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570976 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570978 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570981 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570983 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570986 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570988 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570991 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:14.584357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.570995 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571006 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571009 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571013 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571018 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571021 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571024 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571026 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571029 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571031 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571034 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571036 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571039 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571042 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571044 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571047 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571050 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571052 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571055 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:14.584815 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571057 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571060 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571062 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571065 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571067 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571070 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571073 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571075 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571078 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571080 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571083 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571085 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571088 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571090 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571094 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571104 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571107 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571111 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571113 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571116 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:14.585303 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571118 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571121 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571123 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571126 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571129 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571131 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571134 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571136 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571140 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571142 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571144 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571147 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571149 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571152 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571155 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571157 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571160 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571162 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571165 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:14.585785 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571168 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571170 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571174 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571176 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571179 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571181 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571184 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571189 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.571191 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.574858 4952 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.574874 4952 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.574890 4952 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.574895 4952 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.574989 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.574994 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.574998 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575000 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575003 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575006 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575009 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:14.586260 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575011 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575014 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575017 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575019 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575022 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575025 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575027 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575030 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575033 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575035 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575038 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575040 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575043 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575046 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575048 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575051 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575053 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575056 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575059 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575063 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:14.586734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575067 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575071 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575075 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575078 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575081 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575084 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575086 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575089 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575091 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575095 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575099 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575102 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575105 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575107 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575110 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575112 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575115 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575117 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575120 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:14.587221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575123 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575126 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575128 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575131 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575133 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575136 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575138 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575141 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575144 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575146 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575149 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575152 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575156 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575158 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575161 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575163 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575166 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575169 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575172 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:14.587696 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575174 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575177 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575180 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575182 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575185 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575188 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575190 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575193 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575195 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575198 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575201 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575203 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575206 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575209 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575211 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575214 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575217 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575219 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575222 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:14.588169 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575224 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575227 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575230 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575232 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575235 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575237 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575240 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575243 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575246 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575248 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575250 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575253 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575256 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575258 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575261 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575263 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575266 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575269 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575272 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575274 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:14.588622 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575276 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575279 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.575306 4952 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.575996 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576010 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576015 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576021 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576026 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576030 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576035 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576039 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576045 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576049 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576061 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576065 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576070 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:14.589194 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576075 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576081 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576086 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576091 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576096 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576101 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576106 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576111 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576121 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576125 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576135 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576140 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576145 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576149 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576154 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576159 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576163 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576167 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576172 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:14.589573 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576176 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576181 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576185 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576190 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576199 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576203 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576207 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576212 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576219 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576226 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576231 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576236 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576241 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576245 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576249 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576254 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576264 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576268 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576305 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576311 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:14.590074 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576317 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576321 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576354 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576359 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576363 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576367 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576371 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576375 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576379 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576389 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576394 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576398 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576402 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576407 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576411 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576416 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576420 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576427 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576432 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:14.590544 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576438 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576443 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576454 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576458 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576462 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576466 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576470 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576474 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576478 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576482 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576486 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576490 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576495 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576499 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576508 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576512 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576596 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576770 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576776 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:14.591021 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576782 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576785 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576788 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576792 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576795 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576797 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576800 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576803 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576806 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576808 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576811 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576814 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576817 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576820 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576822 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576825 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:14.591540 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.576862 4952 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576920 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576926 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576929 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576933 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576935 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576938 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576941 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576943 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576947 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576950 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576953 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576955 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576958 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576961 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576963 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576966 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576969 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576971 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576974 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 21:42:14.591950 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576977 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576980 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576982 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576985 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576987 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576990 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576993 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576995 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.576998 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577000 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577003 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577005 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577008 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577011 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577013 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577016 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577020 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577025 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577028 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 21:42:14.592433 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577031 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577034 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577037 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577040 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577042 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577045 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577048 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577051 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577053 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577057 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577060 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577062 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577065 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577068 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577071 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577073 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577076 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577079 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577082 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577085 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 21:42:14.592889 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577088 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577090 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577094 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577097 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577100 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577103 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577107 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577111 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577114 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577116 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577119 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577122 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577124 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577127 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577130 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577133 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577136 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577140 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577143 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 21:42:14.593456 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577145 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577149 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577151 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577154 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577156 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577159 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577162 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577165 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577168 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577170 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577173 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577176 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577178 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577181 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577184 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577186 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577190 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577193 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577197 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577199 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 21:42:14.594241 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577202 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577204 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577207 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577210 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577212 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577215 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577217 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577220 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:14.577223 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.577279 4952 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.578248 4952 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.578299 4952 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.578473 4952 server.go:1014] "Starting client certificate rotation" Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.578617 4952 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.578659 4952 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:36:34 +0000 UTC" deadline="2028-04-04 09:13:18.072029555 +0000 UTC" Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.578685 4952 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12539h31m3.493346546s" Oct 29 21:42:14.595024 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.578895 4952 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:42:14.595716 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.580015 4952 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 21:42:14.595716 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.581254 4952 log.go:25] "Validated CRI v1 runtime API" Oct 29 21:42:14.595716 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.582857 4952 log.go:25] "Validated CRI v1 image API" Oct 29 21:42:14.595716 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.583616 4952 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 21:42:14.595716 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.585413 4952 fs.go:135] Filesystem UUIDs: map[3e918413-df37-4169-9930-ad24e962ebf7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d2d80b37-64ed-4dd3-a50b-a65dac636611:/dev/nvme0n1p3] Oct 29 21:42:14.596216 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.585432 4952 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0024dfc8c86490a909c81911a36e9e55a1b30073969387326eb0b7f225852080/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0024dfc8c86490a909c81911a36e9e55a1b30073969387326eb0b7f225852080/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0eda55ea5d4453f536acf3538eac8378674a1e52ee5ae524cb7bfdb62b528a10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0eda55ea5d4453f536acf3538eac8378674a1e52ee5ae524cb7bfdb62b528a10/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1643973be0e38523f3b9cd7c13f4c66a87c0a4f3af1abb9493379cacd093a2b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1643973be0e38523f3b9cd7c13f4c66a87c0a4f3af1abb9493379cacd093a2b5/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b3c6dd04afb96255e3827b95c08c4ffe5059c6b36948b812223e700619d59fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b3c6dd04afb96255e3827b95c08c4ffe5059c6b36948b812223e700619d59fa/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21a7875621d9fd575976eb9219c5bcb0bc9c53f3a294f76732386708a875a1e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21a7875621d9fd575976eb9219c5bcb0bc9c53f3a294f76732386708a875a1e3/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4497a77913293bef0400eae4eea98a88726ab3a897a4669a8c046458f8fe4a0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4497a77913293bef0400eae4eea98a88726ab3a897a4669a8c046458f8fe4a0c/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44f6a8de86887cbdb030477928904527a0b60058ca463e571005572d449fa744/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44f6a8de86887cbdb030477928904527a0b60058ca463e571005572d449fa744/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d61eeb039f80edf4e1629e9d376fcc83534a34d8e011f58f3225cf5ae27994a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d61eeb039f80edf4e1629e9d376fcc83534a34d8e011f58f3225cf5ae27994a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71c090c7e7270dc7cc8a18b9826c3d50fcfa30d9634863c2f555d2c61f057f15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71c090c7e7270dc7cc8a18b9826c3d50fcfa30d9634863c2f555d2c61f057f15/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71ee871b237a77d68af20d3963de811db9c0123270ca5ba9aaa071d830bef998/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71ee871b237a77d68af20d3963de811db9c0123270ca5ba9aaa071d830bef998/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/737395ecc4418578cf52b2ba4f18a331d771cbc7a0d576db82815abf68e66f6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/737395ecc4418578cf52b2ba4f18a331d771cbc7a0d576db82815abf68e66f6b/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a6f507bbd09eb4724da3ed66ffd47c18814cc9e145f2c5ecbc8bcd7c7224fe8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a6f507bbd09eb4724da3ed66ffd47c18814cc9e145f2c5ecbc8bcd7c7224fe8/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9371a4bc96e83be171ae0fdacfe6791bde2e3aedc040569459d08e98062da97e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9371a4bc96e83be171ae0fdacfe6791bde2e3aedc040569459d08e98062da97e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd32bbcea0e44cba9598bff7d80e116317a22a9369a9baccdf142a05d35b5211/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd32bbcea0e44cba9598bff7d80e116317a22a9369a9baccdf142a05d35b5211/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf5bf574f821b4f7acabc9a2da7906642ebc55e0677f7310a51b36ecdb56a350/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf5bf574f821b4f7acabc9a2da7906642ebc55e0677f7310a51b36ecdb56a350/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4050a5c505d4eb0544fc2048219393ba857834c1918d2af92f4d522f779e95e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4050a5c505d4eb0544fc2048219393ba857834c1918d2af92f4d522f779e95e/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c47d9ccf1cd78d617c085e1aac96743be4175a29223f17b2ac31a3de366b2d06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c47d9ccf1cd78d617c085e1aac96743be4175a29223f17b2ac31a3de366b2d06/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cff6966fdb10b6799cd5ef1268fe57b84f35ce5f0d450ee04156ff27af404ca3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cff6966fdb10b6799cd5ef1268fe57b84f35ce5f0d450ee04156ff27af404ca3/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1db0e66a80973c5eaba40d1897880bbd7e3e5a886b4ec23d97fc3cdd76d6b5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1db0e66a80973c5eaba40d1897880bbd7e3e5a886b4ec23d97fc3cdd76d6b5c/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc552fd029be282c995addbb015fc26b60f5e32ad52197dc897acbc1048c9473/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc552fd029be282c995addbb015fc26b60f5e32ad52197dc897acbc1048c9473/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e349b10ee8d9a8abebdd54178777d129826acf750219dec77bee8286311d500e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e349b10ee8d9a8abebdd54178777d129826acf750219dec77bee8286311d500e/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e81dd74f03f2acbdef9d82f844a6d94f8fe760bcde0c91a1cfbe2853d6c1c020/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e81dd74f03f2acbdef9d82f844a6d94f8fe760bcde0c91a1cfbe2853d6c1c020/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4357f1988c399aa3fb5b008a5f7c483767e42a0414013edf71baa45ca89b904/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4357f1988c399aa3fb5b008a5f7c483767e42a0414013edf71baa45ca89b904/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/059a06d7-5255-47dc-8ce7-07d113e03ca1/volumes/kubernetes.io~projected/kube-api-access-887lk:{mountpoint:/var/lib/kubelet/pods/059a06d7-5255-47dc-8ce7-07d113e03ca1/volumes/kubernetes.io~projected/kube-api-access-887lk major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/059a06d7-5255-47dc-8ce7-07d113e03ca1/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/059a06d7-5255-47dc-8ce7-07d113e03ca1/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/059a06d7-5255-47dc-8ce7-07d113e03ca1/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/059a06d7-5255-47dc-8ce7-07d113e03ca1/volumes/kubernetes.io~secret/stats-auth major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1461042e-6499-452c-b548-cb243c31b17d/volumes/kubernetes.io~projected/kube-api-access-6qnfp:{mountpoint:/var/lib/kubelet/pods/1461042e-6499-452c-b548-cb243c31b17d/volumes/kubernetes.io~projected/kube-api-access-6qnfp major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1461042e-6499-452c-b548-cb243c31b17d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1461042e-6499-452c-b548-cb243c31b17d/volumes/kubernetes.io~secret/cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a8cd583-ba4f-4f3a-9f42-f43d63462d32/volumes/kubernetes.io~projected/kube-api-access-qr9fw:{mountpoint:/var/lib/kubelet/pods/1a8cd583-ba4f-4f3a-9f42-f43d63462d32/volumes/kubernetes.io~projected/kube-api-access-qr9fw major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b583a61-cc48-4c4e-9e2e-7f6e684d45e8/volumes/kubernetes.io~projected/kube-api-access-qzv72:{mountpoint:/var/lib/kubelet/pods/1b583a61-cc48-4c4e-9e2e-7f6e684d45e8/volumes/kubernetes.io~projected/kube-api-access-qzv72 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f4b9c11-19cd-46b9-8096-5c9375fa7e91/volumes/kubernetes.io~projected/kube-api-access-7xdxh:{mountpoint:/var/lib/kubelet/pods/1f4b9c11-19cd-46b9-8096-5c9375fa7e91/volumes/kubernetes.io~projected/kube-api-access-7xdxh major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f4b9c11-19cd-46b9-8096-5c9375fa7e91/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1f4b9c11-19cd-46b9-8096-5c9375fa7e91/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20070f80-4b65-40ff-9646-e8b7fef85cc1/volumes/kubernetes.io~projected/kube-api-access-xgrjk:{mountpoint:/var/lib/kubelet/pods/20070f80-4b65-40ff-9646-e8b7fef85cc1/volumes/kubernetes.io~projected/kube-api-access-xgrjk major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20070f80-4b65-40ff-9646-e8b7fef85cc1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/20070f80-4b65-40ff-9646-e8b7fef85cc1/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33291806-5f11-4f13-ba59-92a143fe8f0e/volumes/kubernetes.io~projected/kube-api-access-5lzfl:{mountpoint:/var/lib/kubelet/pods/33291806-5f11-4f13-ba59-92a143fe8f0e/volumes/kubernetes.io~projected/kube-api-access-5lzfl major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3398371c-c328-4876-963c-8af098bae440/volumes/kubernetes.io~projected/kube-api-access-rsvhn:{mountpoint:/var/lib/kubelet/pods/3398371c-c328-4876-963c-8af098bae440/volumes/kubernetes.io~projected/kube-api-access-rsvhn major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3dbda061-df54-482d-96c6-851f1acf58a4/volumes/kubernetes.io~projected/kube-api-access-zknj5:{mountpoint:/var/lib/kubelet/pods/3dbda061-df54-482d-96c6-851f1acf58a4/volumes/kubernetes.io~projected/kube-api-access-zknj5 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3dd58b31-fa23-4a8c-b113-ad9479a7b681/volumes/kubernetes.io~projected/kube-api-access-2kflk:{mountpoint:/var/lib/kubelet/pods/3dd58b31-fa23-4a8c-b113-ad9479a7b681/volumes/kubernetes.io~projected/kube-api-access-2kflk major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4eb95095-a274-43e2-adbd-d88824033bb1/volumes/kubernetes.io~projected/kube-api-access-mgtcq:{mountpoint:/var/lib/kubelet/pods/4eb95095-a274-43e2-adbd-d88824033bb1/volumes/kubernetes.io~projected/kube-api-access-mgtcq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53ff731b-5c02-4366-b6b2-c5ecfdc8be38/volumes/kubernetes.io~projected/kube-api-access-mngv9:{mountpoint:/var/lib/kubelet/pods/53ff731b-5c02-4366-b6b2-c5ecfdc8be38/volumes/kubernetes.io~projected/kube-api-access-mngv9 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587e4eb6-4e54-4c6e-a640-71976f13827c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/587e4eb6-4e54-4c6e-a640-71976f13827c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587e4eb6-4e54-4c6e-a640-71976f13827c/volumes/kubernetes.io~projected/kube-api-access-r9hc6:{mountpoint:/var/lib/kubelet/pods/587e4eb6-4e54-4c6e-a640-71976f13827c/volumes/kubernetes.io~projected/kube-api-access-r9hc6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587e4eb6-4e54-4c6e-a640-71976f13827c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/587e4eb6-4e54-4c6e-a640-71976f13827c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~projected/kube-api-access-qhssb:{mountpoint:/var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~projected/kube-api-access-qhssb major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61ac615e-a093-42ea-92a4-84d5c22c9f41/volumes/kubernetes.io~projected/kube-api-access-zj6l4:{mountpoint:/var/lib/kubelet/pods/61ac615e-a093-42ea-92a4-84d5c22c9f41/volumes/kubernetes.io~projected/kube-api-access-zj6l4 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61ac615e-a093-42ea-92a4-84d5c22c9f41/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/61ac615e-a093-42ea-92a4-84d5c22c9f41/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73219cee-7208-4f68-ad0b-bc9cb0bcb199/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/73219cee-7208-4f68-ad0b-bc9cb0bcb199/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84058be2-867e-4c5a-a614-bccfd7af96fa/volumes/kubernetes.io~projected/kube-api-access-6kjpx:{mountpoint:/var/lib/kubelet/pods/84058be2-867e-4c5a-a614-bccfd7af96fa/volumes/kubernetes.io~projected/kube-api-access-6kjpx major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88711e24-d089-4fbb-9b36-fb015fe7f9c9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/88711e24-d089-4fbb-9b36-fb015fe7f9c9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88711e24-d089-4fbb-9b36-fb015fe7f9c9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/88711e24-d089-4fbb-9b36-fb015fe7f9c9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88711e24-d089-4fbb-9b36-fb015fe7f9c9/volumes/kubernetes.io~projected/kube-api-access-bdmgp:{mountpoint:/var/lib/kubelet/pods/88711e24-d089-4fbb-9b36-fb015fe7f9c9/volumes/kubernetes.io~projected/kube-api-access-bdmgp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89de847b-bcca-4f80-af6b-071681d3afdb/volumes/kubernetes.io~projected/kube-api-access-bn95n:{mountpoint:/var/lib/kubelet/pods/89de847b-bcca-4f80-af6b-071681d3afdb/volumes/kubernetes.io~projected/kube-api-access-bn95n major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~projected/kube-api-access-wv7n9:{mountpoint:/var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~projected/kube-api-access-wv7n9 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97611cb7-e267-4d11-b20a-c52d833cdefc/volumes/kubernetes.io~projected/kube-api-access-bhmp9:{mountpoint:/var/lib/kubelet/pods/97611cb7-e267-4d11-b20a-c52d833cdefc/volumes/kubernetes.io~projected/kube-api-access-bhmp9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4dbf16f-ae81-461b-8564-99fce7202e3c/volumes/kubernetes.io~projected/kube-api-access-qg6zw:{mountpoint:/var/lib/kubelet/pods/a4dbf16f-ae81-461b-8564-99fce7202e3c/volumes/kubernetes.io~projected/kube-api-access-qg6zw major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b39d65b2-cca6-4f08-bdec-becd6e000c48/volumes/kubernetes.io~projected/kube-api-access-kcgk9:{mountpoint:/var/lib/kubelet/pods/b39d65b2-cca6-4f08-bdec-becd6e000c48/volumes/kubernetes.io~projected/kube-api-access-kcgk9 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9/volumes/kubernetes.io~projected/kube-api-access-7x8ph:{mountpoint:/var/lib/kubelet/pods/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9/volumes/kubernetes.io~projected/kube-api-access-7x8ph major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcf1d737-3cce-4759-8bde-21c7e0dff0ae/volumes/kubernetes.io~projected/kube-api-access-rzw9z:{mountpoint:/var/lib/kubelet/pods/dcf1d737-3cce-4759-8bde-21c7e0dff0ae/volumes/kubernetes.io~projected/kube-api-access-rzw9z major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8/volumes/kubernetes.io~projected/kube-api-access-q2nfl:{mountpoint:/var/lib/kubelet/pods/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8/volumes/kubernetes.io~projected/kube-api-access-q2nfl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec6efbe6-a797-4508-a997-c1b3aceec331/volumes/kubernetes.io~projected/kube-api-access-v9bql:{mountpoint:/var/lib/kubelet/pods/ec6efbe6-a797-4508-a997-c1b3aceec331/volumes/kubernetes.io~projected/kube-api-access-v9bql major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f04d5d43-128a-4972-b55e-836175e60130/volumes/kubernetes.io~projected/kube-api-access-m9h8r:{mountpoint:/var/lib/kubelet/pods/f04d5d43-128a-4972-b55e-836175e60130/volumes/kubernetes.io~projected/kube-api-access-m9h8r major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8539056-ee01-4262-89b4-1d4b55a5073c/volumes/kubernetes.io~projected/kube-api-access-x4k74:{mountpoint:/var/lib/kubelet/pods/f8539056-ee01-4262-89b4-1d4b55a5073c/volumes/kubernetes.io~projected/kube-api-access-x4k74 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb5b0077-ddac-46d3-ba06-5fc065be5b53/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fb5b0077-ddac-46d3-ba06-5fc065be5b53/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cad1c7d8c64d5fa0db9417b2251880b7c8e9a54842ee21bb35a35c7cfd928d47/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/75ad9fbf30dd784c9f4b2cc79abdf96c6781729e7fb1d422f2f5cc51d9d4b3dc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e08e143a2b276fa2fef353c36223aa17e2c3d8ae7660b7961998ede412705c70/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ccb9d524441d15cfd9e048a693b3d305354e0cdaf64b45880edf0c24aca24277/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d0867db1b96e82a0e571d93ba71c4f6f9de40c02ee4546d8ce8bbaeaaaf026cd/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e2067b5dbd698275c3f4225d5179531790f32c708155c488ea6f9f365bc41f34/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5a48a248f9fbda12c1efdaa4a4ac4175c632c7d3a828f080ff8f5fc0ee860784/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9fac19b6d3c63a8a8513469886c21be578aa2fa781224a50dec3bb73bd712870/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/57cc6cd270525bdbf865424dfd5c9f330f3c56613460709d5b0d1c2f6042af1b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/682cf7902840d7bd29e5293c14b8f54528fdddd24713c3855ffc005ce952616f/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/aafa1140d3e0ca199718f61db5d06786075fddd19400687ff937722bb16f15d1/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e06178ed50b4699fa2a230a9445af7dcf497adcb4049958482e48c5d1578f4a4/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1c00ec9205e7bcf60181d82ad61b77170f0d005d1c4cf9e1276163d1cc2c6851/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0da5938526a975ab4db1d86382855d4ce1f9b414029d9fec20324e23893bd339/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/75e746bf028fe05e17e4d04fdbce573871ebc7bfac93114fef2f95f49593e382/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0add001dc919aae366ddeb69fe4ef95bc0ac9f3caa66d3e7245a36c5b43d3405/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b1aefa3f460d3ba405208066518a893ee59af5a6ebe9a1bcdc7e304dcebceba9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/528546f4d4d24d388064a7e365f0f84336abea25c6adaaacb3b89d8eac2a1914/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/57f9082dd7ddcab71e370d5e94687a4ec41b12c61e74a85b386bb651df2e8746/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cdb9753dafd0323ee4b18bb87b45f427ba3a01e2233152a157500bc531851c66/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/39d59fad5d5790ee78bd14a9320e730e32ba6dfb02441f8258cc3df9745692c4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/83f86f551fd5b86b58c294e6502cc8747b062c5ee33b252e4f0e7c4afe270769/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/6c794d6c5ef7cfdb9a96cdafc073a623600aeba72283acba3751410f3eab641f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/9f8ea333f3cbd443e6a4a2708f0bbb062e50712020e1e900a236e24f3831bfdc/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/fc65186ac601441867200ab9a99792abf95fe11b9537c91f42137c6b1fe9b7c9/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/1cd693836df51ac77a8b855bf8f6cdcd35761c706bcf4a96899ba065da945699/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/5a721ca3515a5fd46ba2bac557cfea24080f7b4f929236ae6d7cec7d081ea436/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/3d3a09e2159701176226943b679d54315d29a1f1a638ecf64995d3c05958e96d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/4aef0c1a31100da1ab3b9a16aa0689a5b3d158bba8a0763147983fb21f5302b5/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/5b95fda1f6fcc9a96e93323262a3199345f9dd6290df97ca11d30a56a2997c07/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/d12a820cd7ff437fe05b0bc74a3e5326ed99ba341e434214e2ab6b3cb40c72a3/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/08d3402ec3aa0df30b7981ca1759b0424fdb08f8e1218de94f69fa5aa52f3d5e/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/b67682ccbe9b6adf9f6b6ea978bf6d183d35c6fcdf54b06ec7414a1a11b9c5fe/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/53ca3b6d56be7f858851d7f25f0674c794413125ae41b5df5ccf7d30461324cb/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/c3f755c2d55c8b3a830df2d2047087af60662ef77206fed9c1d2575463f2b2ff/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/54c6bb93f0f6f256e28cbd98450137d507fd8979fddaf81e1b1b31c225df87d1/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/c1c947d7d73677d2188d74d23ad3583940cf52d79acc3aa454bc837ed86d28e6/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/97b5ca0f7fc0dd8448a9f3e12f964b82a2cd01669e17613cf919cc77ceb2c004/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/3237b8f6503a26f35a59bf7762c3664143f8829d7ea35a838ab0124cfa0e2f78/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/b3f95d8c679ed9fd449fccfde64fdf28648c7bc4031a47767dc947c84b0ff105/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/cc6a2604daf5f90158d92d0b17b5504412638babf81dddeab498472ea33d5908/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/48f10565f597c16dcdf4afe95a81c7d177542fb97befcc151911c7c3ced91ff3/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/85f0a09e252bbf42a05d581686d3b485838982a9d0dd5131338a0f9045f39ac2/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/ebe85394083f6700e06b15fbb976921e88638b437af539da717a6c42b6854bca/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/a45af44eb0e73f083e9ce8809203caf33d755cdad7122df2b68c761948dd0f0c/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/4b0c6ca1c9aa4583b5e639d3eb1c448769076a496b4142443668634f024ee1a8/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/219da0a49014e0d87eba796aec657c08d063660e3d73dbba4f1cf44350332182/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/4222826c282c54e33ee7548c9df3217f7259ac4d3c1920e2c5fc872da3348235/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/57a5c8e229819cda38bb3cbc7ebeefb7932c3139fae950f4c4f1604c04af6eec/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/311bf22f9b81547ac04d99ffacada134d6f187b5260b3b8ab841019562357451/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4795975e478bc39c7a3fdf094ac32b8eb4374176e1912319145bfda4c2071267/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/baf1e22e2bfea5d8d88ebd0a2883acede55fe85a7aec852b62124acc95e66c36/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9ea914b3f6463998cda9105b0782ebde0237a435ec8a18e58a35ca646eddfc04/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0c36070a38cd04beef7d45d0eafb5c486853a2e3402db59ea5813099eab6132f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/527289285e67ce000235abf97035cf08aa7c8b9498eefa798b13fbd697af602f/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/fcb4a879bdf5aa24c6098a8f86bef30543391ea74b8735eb1909a1f793c1a16d/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.592870 4952 manager.go:217] Machine: {Timestamp:2026-10-29 21:42:14.59240262 +0000 UTC m=+0.066991029 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099650 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29ed2a1f28fece6c94604c082cce54 SystemUUID:ec29ed2a-1f28-fece-6c94-604c082cce54 BootID:e432c050-bb27-4ac2-beeb-5ee9b17b693b Filesystems:[{Device:/var/lib/kubelet/pods/4eb95095-a274-43e2-adbd-d88824033bb1/volumes/kubernetes.io~projected/kube-api-access-mgtcq DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1a8cd583-ba4f-4f3a-9f42-f43d63462d32/volumes/kubernetes.io~projected/kube-api-access-qr9fw DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a6f507bbd09eb4724da3ed66ffd47c18814cc9e145f2c5ecbc8bcd7c7224fe8/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc552fd029be282c995addbb015fc26b60f5e32ad52197dc897acbc1048c9473/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3dbda061-df54-482d-96c6-851f1acf58a4/volumes/kubernetes.io~projected/kube-api-access-zknj5 DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec6efbe6-a797-4508-a997-c1b3aceec331/volumes/kubernetes.io~projected/kube-api-access-v9bql DeviceMajor:0 DeviceMinor:335 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9371a4bc96e83be171ae0fdacfe6791bde2e3aedc040569459d08e98062da97e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd32bbcea0e44cba9598bff7d80e116317a22a9369a9baccdf142a05d35b5211/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4497a77913293bef0400eae4eea98a88726ab3a897a4669a8c046458f8fe4a0c/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c47d9ccf1cd78d617c085e1aac96743be4175a29223f17b2ac31a3de366b2d06/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f04d5d43-128a-4972-b55e-836175e60130/volumes/kubernetes.io~projected/kube-api-access-m9h8r DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf5bf574f821b4f7acabc9a2da7906642ebc55e0677f7310a51b36ecdb56a350/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73219cee-7208-4f68-ad0b-bc9cb0bcb199/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20070f80-4b65-40ff-9646-e8b7fef85cc1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/059a06d7-5255-47dc-8ce7-07d113e03ca1/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3dd58b31-fa23-4a8c-b113-ad9479a7b681/volumes/kubernetes.io~projected/kube-api-access-2kflk DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44f6a8de86887cbdb030477928904527a0b60058ca463e571005572d449fa744/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/88711e24-d089-4fbb-9b36-fb015fe7f9c9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b583a61-cc48-4c4e-9e2e-7f6e684d45e8/volumes/kubernetes.io~projected/kube-api-access-qzv72 DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21a7875621d9fd575976eb9219c5bcb0bc9c53f3a294f76732386708a875a1e3/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/88711e24-d089-4fbb-9b36-fb015fe7f9c9/volumes/kubernetes.io~projected/kube-api-access-bdmgp DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61ac615e-a093-42ea-92a4-84d5c22c9f41/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/97611cb7-e267-4d11-b20a-c52d833cdefc/volumes/kubernetes.io~projected/kube-api-access-bhmp9 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/737395ecc4418578cf52b2ba4f18a331d771cbc7a0d576db82815abf68e66f6b/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1db0e66a80973c5eaba40d1897880bbd7e3e5a886b4ec23d97fc3cdd76d6b5c/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcf1d737-3cce-4759-8bde-21c7e0dff0ae/volumes/kubernetes.io~projected/kube-api-access-rzw9z DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/587e4eb6-4e54-4c6e-a640-71976f13827c/volumes/kubernetes.io~projected/kube-api-access-r9hc6 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b39d65b2-cca6-4f08-bdec-becd6e000c48/volumes/kubernetes.io~projected/kube-api-access-kcgk9 DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8/volumes/kubernetes.io~projected/kube-api-access-q2nfl DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d61eeb039f80edf4e1629e9d376fcc83534a34d8e011f58f3225cf5ae27994a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e81dd74f03f2acbdef9d82f844a6d94f8fe760bcde0c91a1cfbe2853d6c1c020/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1643973be0e38523f3b9cd7c13f4c66a87c0a4f3af1abb9493379cacd093a2b5/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f4b9c11-19cd-46b9-8096-5c9375fa7e91/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e349b10ee8d9a8abebdd54178777d129826acf750219dec77bee8286311d500e/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0eda55ea5d4453f536acf3538eac8378674a1e52ee5ae524cb7bfdb62b528a10/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~projected/kube-api-access-qhssb DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88711e24-d089-4fbb-9b36-fb015fe7f9c9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/587e4eb6-4e54-4c6e-a640-71976f13827c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61ac615e-a093-42ea-92a4-84d5c22c9f41/volumes/kubernetes.io~projected/kube-api-access-zj6l4 DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53ff731b-5c02-4366-b6b2-c5ecfdc8be38/volumes/kubernetes.io~projected/kube-api-access-mngv9 DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cff6966fdb10b6799cd5ef1268fe57b84f35ce5f0d450ee04156ff27af404ca3/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0024dfc8c86490a909c81911a36e9e55a1b30073969387326eb0b7f225852080/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/059a06d7-5255-47dc-8ce7-07d113e03ca1/volumes/kubernetes.io~projected/kube-api-access-887lk DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71c090c7e7270dc7cc8a18b9826c3d50fcfa30d9634863c2f555d2c61f057f15/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/20070f80-4b65-40ff-9646-e8b7fef85cc1/volumes/kubernetes.io~projected/kube-api-access-xgrjk DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b3c6dd04afb96255e3827b95c08c4ffe5059c6b36948b812223e700619d59fa/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4357f1988c399aa3fb5b008a5f7c483767e42a0414013edf71baa45ca89b904/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/89de847b-bcca-4f80-af6b-071681d3afdb/volumes/kubernetes.io~projected/kube-api-access-bn95n DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/059a06d7-5255-47dc-8ce7-07d113e03ca1/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1f4b9c11-19cd-46b9-8096-5c9375fa7e91/volumes/kubernetes.io~projected/kube-api-access-7xdxh DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9/volumes/kubernetes.io~projected/kube-api-access-7x8ph DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4dbf16f-ae81-461b-8564-99fce7202e3c/volumes/kubernetes.io~projected/kube-api-access-qg6zw DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4050a5c505d4eb0544fc2048219393ba857834c1918d2af92f4d522f779e95e/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/33291806-5f11-4f13-ba59-92a143fe8f0e/volumes/kubernetes.io~projected/kube-api-access-5lzfl DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes/kubernetes.io~projected/kube-api-access-wv7n9 DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1461042e-6499-452c-b548-cb243c31b17d/volumes/kubernetes.io~projected/kube-api-access-6qnfp DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/587e4eb6-4e54-4c6e-a640-71976f13827c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3398371c-c328-4876-963c-8af098bae440/volumes/kubernetes.io~projected/kube-api-access-rsvhn DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb5b0077-ddac-46d3-ba06-5fc065be5b53/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/84058be2-867e-4c5a-a614-bccfd7af96fa/volumes/kubernetes.io~projected/kube-api-access-6kjpx DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1461042e-6499-452c-b548-cb243c31b17d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71ee871b237a77d68af20d3963de811db9c0123270ca5ba9aaa071d830bef998/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8539056-ee01-4262-89b4-1d4b55a5073c/volumes/kubernetes.io~projected/kube-api-access-x4k74 DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1643973be0e3852 MacAddress:36:b2:fa:5c:59:5d Speed:10000 Mtu:8901} {Name:1b3c6dd04afb962 MacAddress:ea:30:f4:69:2b:c2 Speed:10000 Mtu:8901} {Name:21a7875621d9fd5 MacAddress:56:2a:91:13:36:07 Speed:10000 Mtu:8901} {Name:4497a77913293be MacAddress:62:af:7e:24:76:45 Speed:10000 Mtu:8901} {Name:71ee871b237a77d MacAddress:8e:7d:98:8f:96:69 Speed:10000 Mtu:8901} {Name:bd32bbcea0e44cb MacAddress:0a:2b:45:89:e7:31 Speed:10000 Mtu:8901} {Name:bf5bf574f821b4f MacAddress:12:93:f0:8e:ff:f4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d2:d1:09:57 Speed:0 Mtu:9001} {Name:br-int MacAddress:e6:a6:a7:25:1a:e5 Speed:0 Mtu:8901} {Name:c4050a5c505d4eb MacAddress:d2:56:59:24:2d:06 Speed:10000 Mtu:8901} {Name:c47d9ccf1cd78d6 MacAddress:66:3f:f8:45:d2:7d Speed:10000 Mtu:8901} {Name:cff6966fdb10b67 MacAddress:6e:68:17:37:ff:ab Speed:10000 Mtu:8901} {Name:d1db0e66a80973c MacAddress:12:d0:b8:47:ab:8b Speed:10000 Mtu:8901} {Name:dc552fd029be282 MacAddress:56:11:30:4a:45:cb Speed:10000 Mtu:8901} {Name:e349b10ee8d9a8a MacAddress:8e:25:c1:41:23:c4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d2:d1:09:57 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:06:6f:65:15:f6:6f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:8d:54:b3:d9:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.594780 4952 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.594847 4952 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595210 4952 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595235 4952 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595416 4952 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595429 4952 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595447 4952 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595467 4952 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595493 4952 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595707 4952 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595846 4952 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595889 4952 kubelet.go:493] "Attempting to sync node with API server" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595926 4952 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595940 4952 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595953 4952 kubelet.go:405] "Adding apiserver pod source" Oct 29 21:42:14.596546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.595964 4952 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 21:42:14.597390 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.596872 4952 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 21:42:14.597390 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.596890 4952 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:42:14.597390 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.597377 4952 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 21:42:14.599612 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.599589 4952 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 21:42:14.599612 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.599611 4952 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 21:42:14.599719 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.599709 4952 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 21:42:14.600006 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.599889 4952 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 21:42:14.600006 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.599944 4952 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 21:42:14.600116 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600041 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 21:42:14.600116 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600058 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 21:42:14.600116 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600068 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 21:42:14.600116 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600092 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 21:42:14.600116 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600106 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 21:42:14.600237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600121 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 21:42:14.600237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600134 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 21:42:14.600237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600144 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 21:42:14.600237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600158 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 21:42:14.600237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600172 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 21:42:14.600237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600197 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 21:42:14.600237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600208 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 21:42:14.600237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600232 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 21:42:14.600446 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.600242 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 21:42:14.603680 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.603667 4952 server.go:1263] "Started kubelet" Oct 29 21:42:14.603774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.603737 4952 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 21:42:14.605039 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.605023 4952 server.go:317] "Adding debug handlers to kubelet server" Oct 29 21:42:14.605182 ip-10-0-130-134 systemd[1]: Started Kubernetes Kubelet. Oct 29 21:42:14.608244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.606153 4952 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 21:42:14.608244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.606452 4952 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 21:42:14.608244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.606806 4952 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 21:42:14.610415 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.610396 4952 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 21:42:14.610495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.610477 4952 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.610624 4952 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.610633 4952 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:36:36 +0000 UTC" deadline="2028-05-08 09:56:33.312682775 +0000 UTC" Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.610649 4952 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.610654 4952 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13356h14m18.702032239s" Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.610642 4952 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:14.611267 4952 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-134.ec2.internal\" not found" Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.611584 4952 factory.go:55] Registering systemd factory Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.611607 4952 factory.go:223] Registration of the systemd container factory successfully Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.612491 4952 factory.go:153] Registering CRI-O factory Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.612507 4952 factory.go:223] Registration of the crio container factory successfully Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.612562 4952 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.612579 4952 factory.go:103] Registering Raw factory Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.612590 4952 manager.go:1199] Started watching for new ooms in manager Oct 29 21:42:14.613239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.613128 4952 manager.go:319] Starting recovery of all containers Oct 29 21:42:14.614000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.613874 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" volumeName="kubernetes.io/empty-dir/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-ca-trust-extracted" seLinuxMountContext="" Oct 29 21:42:14.614000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.613932 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97611cb7-e267-4d11-b20a-c52d833cdefc" volumeName="kubernetes.io/projected/97611cb7-e267-4d11-b20a-c52d833cdefc-kube-api-access-bhmp9" seLinuxMountContext="" Oct 29 21:42:14.614000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.613947 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9" volumeName="kubernetes.io/configmap/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-config" seLinuxMountContext="" Oct 29 21:42:14.614000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.613961 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9" volumeName="kubernetes.io/projected/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-kube-api-access-7x8ph" seLinuxMountContext="" Oct 29 21:42:14.614000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.613981 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587e4eb6-4e54-4c6e-a640-71976f13827c" volumeName="kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-env-overrides" seLinuxMountContext="" Oct 29 21:42:14.614000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.613996 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61ac615e-a093-42ea-92a4-84d5c22c9f41" volumeName="kubernetes.io/configmap/61ac615e-a093-42ea-92a4-84d5c22c9f41-config" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614010 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f12b29-bfab-4407-8cae-b618a658c23e" volumeName="kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-bound-sa-token" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614026 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f4b9c11-19cd-46b9-8096-5c9375fa7e91" volumeName="kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-tmp" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614041 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1461042e-6499-452c-b548-cb243c31b17d" volumeName="kubernetes.io/secret/1461042e-6499-452c-b548-cb243c31b17d-cert" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614054 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20070f80-4b65-40ff-9646-e8b7fef85cc1" volumeName="kubernetes.io/secret/20070f80-4b65-40ff-9646-e8b7fef85cc1-serving-cert" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614067 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" volumeName="kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-image-registry-private-configuration" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614080 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97611cb7-e267-4d11-b20a-c52d833cdefc" volumeName="kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-binary-copy" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614094 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eb95095-a274-43e2-adbd-d88824033bb1" volumeName="kubernetes.io/projected/4eb95095-a274-43e2-adbd-d88824033bb1-kube-api-access-mgtcq" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614108 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53ff731b-5c02-4366-b6b2-c5ecfdc8be38" volumeName="kubernetes.io/configmap/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-config" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614123 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587e4eb6-4e54-4c6e-a640-71976f13827c" volumeName="kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-script-lib" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614136 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" volumeName="kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-installation-pull-secrets" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614148 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9" volumeName="kubernetes.io/secret/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-serving-cert" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614164 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec6efbe6-a797-4508-a997-c1b3aceec331" volumeName="kubernetes.io/projected/ec6efbe6-a797-4508-a997-c1b3aceec331-kube-api-access-v9bql" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614176 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88711e24-d089-4fbb-9b36-fb015fe7f9c9" volumeName="kubernetes.io/projected/88711e24-d089-4fbb-9b36-fb015fe7f9c9-kube-api-access-bdmgp" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614188 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f12b29-bfab-4407-8cae-b618a658c23e" volumeName="kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-installation-pull-secrets" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614199 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="059a06d7-5255-47dc-8ce7-07d113e03ca1" volumeName="kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-default-certificate" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614211 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a8cd583-ba4f-4f3a-9f42-f43d63462d32" volumeName="kubernetes.io/projected/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-kube-api-access-qr9fw" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614223 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e889f05-b993-4423-9b9c-4673ad6e70d3" volumeName="kubernetes.io/configmap/1e889f05-b993-4423-9b9c-4673ad6e70d3-nginx-conf" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614235 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dd58b31-fa23-4a8c-b113-ad9479a7b681" volumeName="kubernetes.io/projected/3dd58b31-fa23-4a8c-b113-ad9479a7b681-kube-api-access-2kflk" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614246 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53ff731b-5c02-4366-b6b2-c5ecfdc8be38" volumeName="kubernetes.io/projected/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-kube-api-access-mngv9" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614277 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33291806-5f11-4f13-ba59-92a143fe8f0e" volumeName="kubernetes.io/projected/33291806-5f11-4f13-ba59-92a143fe8f0e-kube-api-access-5lzfl" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614287 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89de847b-bcca-4f80-af6b-071681d3afdb" volumeName="kubernetes.io/configmap/89de847b-bcca-4f80-af6b-071681d3afdb-serviceca" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614312 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8539056-ee01-4262-89b4-1d4b55a5073c" volumeName="kubernetes.io/projected/f8539056-ee01-4262-89b4-1d4b55a5073c-kube-api-access-x4k74" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614333 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" volumeName="kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-trusted-ca" seLinuxMountContext="" Oct 29 21:42:14.614330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614344 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73219cee-7208-4f68-ad0b-bc9cb0bcb199" volumeName="kubernetes.io/secret/73219cee-7208-4f68-ad0b-bc9cb0bcb199-agent-certs" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614356 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88711e24-d089-4fbb-9b36-fb015fe7f9c9" volumeName="kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-tmp" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614368 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4dbf16f-ae81-461b-8564-99fce7202e3c" volumeName="kubernetes.io/projected/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-api-access-qg6zw" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614379 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8" volumeName="kubernetes.io/projected/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-kube-api-access-q2nfl" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614397 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f4b9c11-19cd-46b9-8096-5c9375fa7e91" volumeName="kubernetes.io/projected/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-kube-api-access-7xdxh" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614415 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f4b9c11-19cd-46b9-8096-5c9375fa7e91" volumeName="kubernetes.io/secret/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-serving-cert" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614429 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20070f80-4b65-40ff-9646-e8b7fef85cc1" volumeName="kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-config" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614440 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="059a06d7-5255-47dc-8ce7-07d113e03ca1" volumeName="kubernetes.io/projected/059a06d7-5255-47dc-8ce7-07d113e03ca1-kube-api-access-887lk" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614452 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a8cd583-ba4f-4f3a-9f42-f43d63462d32" volumeName="kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-daemon-config" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614464 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97611cb7-e267-4d11-b20a-c52d833cdefc" volumeName="kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614490 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39d65b2-cca6-4f08-bdec-becd6e000c48" volumeName="kubernetes.io/configmap/b39d65b2-cca6-4f08-bdec-becd6e000c48-config-volume" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614508 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20070f80-4b65-40ff-9646-e8b7fef85cc1" volumeName="kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-trusted-ca" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614521 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33291806-5f11-4f13-ba59-92a143fe8f0e" volumeName="kubernetes.io/configmap/33291806-5f11-4f13-ba59-92a143fe8f0e-telemetry-config" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614534 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eb95095-a274-43e2-adbd-d88824033bb1" volumeName="kubernetes.io/configmap/4eb95095-a274-43e2-adbd-d88824033bb1-iptables-alerter-script" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614547 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587e4eb6-4e54-4c6e-a640-71976f13827c" volumeName="kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-config" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614558 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f12b29-bfab-4407-8cae-b618a658c23e" volumeName="kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-kube-api-access-wv7n9" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614572 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f12b29-bfab-4407-8cae-b618a658c23e" volumeName="kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-image-registry-private-configuration" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614588 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97611cb7-e267-4d11-b20a-c52d833cdefc" volumeName="kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614599 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f04d5d43-128a-4972-b55e-836175e60130" volumeName="kubernetes.io/projected/f04d5d43-128a-4972-b55e-836175e60130-kube-api-access-m9h8r" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614613 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" volumeName="kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-bound-sa-token" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614625 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61ac615e-a093-42ea-92a4-84d5c22c9f41" volumeName="kubernetes.io/projected/61ac615e-a093-42ea-92a4-84d5c22c9f41-kube-api-access-zj6l4" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614638 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61ac615e-a093-42ea-92a4-84d5c22c9f41" volumeName="kubernetes.io/secret/61ac615e-a093-42ea-92a4-84d5c22c9f41-serving-cert" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614652 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84058be2-867e-4c5a-a614-bccfd7af96fa" volumeName="kubernetes.io/projected/84058be2-867e-4c5a-a614-bccfd7af96fa-kube-api-access-6kjpx" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614667 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5b0077-ddac-46d3-ba06-5fc065be5b53" volumeName="kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614679 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f4b9c11-19cd-46b9-8096-5c9375fa7e91" volumeName="kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-trusted-ca-bundle" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614690 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89de847b-bcca-4f80-af6b-071681d3afdb" volumeName="kubernetes.io/projected/89de847b-bcca-4f80-af6b-071681d3afdb-kube-api-access-bn95n" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614701 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f12b29-bfab-4407-8cae-b618a658c23e" volumeName="kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-registry-certificates" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614720 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39d65b2-cca6-4f08-bdec-becd6e000c48" volumeName="kubernetes.io/projected/b39d65b2-cca6-4f08-bdec-becd6e000c48-kube-api-access-kcgk9" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614731 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20070f80-4b65-40ff-9646-e8b7fef85cc1" volumeName="kubernetes.io/projected/20070f80-4b65-40ff-9646-e8b7fef85cc1-kube-api-access-xgrjk" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614743 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f4b9c11-19cd-46b9-8096-5c9375fa7e91" volumeName="kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-snapshots" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614755 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a8cd583-ba4f-4f3a-9f42-f43d63462d32" volumeName="kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cni-binary-copy" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614766 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b583a61-cc48-4c4e-9e2e-7f6e684d45e8" volumeName="kubernetes.io/projected/1b583a61-cc48-4c4e-9e2e-7f6e684d45e8-kube-api-access-qzv72" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614776 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f4b9c11-19cd-46b9-8096-5c9375fa7e91" volumeName="kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-service-ca-bundle" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614787 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73219cee-7208-4f68-ad0b-bc9cb0bcb199" volumeName="kubernetes.io/configmap/73219cee-7208-4f68-ad0b-bc9cb0bcb199-konnectivity-ca" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614800 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcf1d737-3cce-4759-8bde-21c7e0dff0ae" volumeName="kubernetes.io/projected/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kube-api-access-rzw9z" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614818 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587e4eb6-4e54-4c6e-a640-71976f13827c" volumeName="kubernetes.io/projected/587e4eb6-4e54-4c6e-a640-71976f13827c-kube-api-access-r9hc6" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614830 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84058be2-867e-4c5a-a614-bccfd7af96fa" volumeName="kubernetes.io/empty-dir/84058be2-867e-4c5a-a614-bccfd7af96fa-tmp-dir" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614842 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88711e24-d089-4fbb-9b36-fb015fe7f9c9" volumeName="kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-tuned" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614853 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39d65b2-cca6-4f08-bdec-becd6e000c48" volumeName="kubernetes.io/empty-dir/b39d65b2-cca6-4f08-bdec-becd6e000c48-tmp-dir" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614865 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3398371c-c328-4876-963c-8af098bae440" volumeName="kubernetes.io/projected/3398371c-c328-4876-963c-8af098bae440-kube-api-access-rsvhn" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614876 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" volumeName="kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-certificates" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614888 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dbda061-df54-482d-96c6-851f1acf58a4" volumeName="kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614916 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" volumeName="kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-kube-api-access-qhssb" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614929 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f12b29-bfab-4407-8cae-b618a658c23e" volumeName="kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-trusted-ca" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614941 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f12b29-bfab-4407-8cae-b618a658c23e" volumeName="kubernetes.io/empty-dir/91f12b29-bfab-4407-8cae-b618a658c23e-ca-trust-extracted" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614952 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4dbf16f-ae81-461b-8564-99fce7202e3c" volumeName="kubernetes.io/empty-dir/a4dbf16f-ae81-461b-8564-99fce7202e3c-data-volume" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614965 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="059a06d7-5255-47dc-8ce7-07d113e03ca1" volumeName="kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-stats-auth" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614979 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1461042e-6499-452c-b548-cb243c31b17d" volumeName="kubernetes.io/projected/1461042e-6499-452c-b548-cb243c31b17d-kube-api-access-6qnfp" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.614993 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587e4eb6-4e54-4c6e-a640-71976f13827c" volumeName="kubernetes.io/secret/587e4eb6-4e54-4c6e-a640-71976f13827c-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.615005 4952 reconstruct.go:97] "Volume reconstruction finished" Oct 29 21:42:14.615479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.615012 4952 reconciler.go:29] "Reconciler: start to sync state" Oct 29 21:42:14.629433 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.629404 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/1.log" Oct 29 21:42:14.629580 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.629479 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/1.log" Oct 29 21:42:14.634207 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.634168 4952 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 21:42:14.635662 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.635640 4952 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 21:42:14.635754 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.635669 4952 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 21:42:14.635754 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.635693 4952 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 21:42:14.635754 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:14.635733 4952 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 21:42:14.655359 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.655335 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/1.log" Oct 29 21:42:14.655456 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.655373 4952 generic.go:356] "Generic (PLEG): container finished" podID="20070f80-4b65-40ff-9646-e8b7fef85cc1" containerID="b66b5d7d234cfd8206a074cdeb2ee9e4cc101586f990e55edee853f3b12fd063" exitCode=255 Oct 29 21:42:14.660257 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.660237 4952 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 21:42:14.660353 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.660269 4952 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:14.661632 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.661614 4952 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:14.674370 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.674350 4952 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:14.674370 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.674358 4952 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 21:42:14.676624 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.676597 4952 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="dde8a0a0304bb3761b89d01f9e1919d7a273276876590433177c8b24e8708291" exitCode=0 Oct 29 21:42:14.676691 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.676625 4952 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="97cfd68c5243f5dbc339e7e335c45d4651d67f1be81610a693817df25579942a" exitCode=0 Oct 29 21:42:14.676691 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.676637 4952 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="4db962e1f813b9d8be9ef98cf91079d62a31918e68a930be2d9759bd36db492c" exitCode=0 Oct 29 21:42:14.676691 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.676647 4952 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="c4c40b580901952aadd901e06e917416155306049624533354264b1d95f6c85d" exitCode=0 Oct 29 21:42:14.676691 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.676657 4952 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="42e055b0bee43840468b0f5f0e72670637ff4b79f196e22a839874a9ae5b39be" exitCode=0 Oct 29 21:42:14.676691 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.676670 4952 generic.go:356] "Generic (PLEG): container finished" podID="97611cb7-e267-4d11-b20a-c52d833cdefc" containerID="77ac20d219726e226b4f910a446380641c5c8b7f86753c4fe4d9fc9ab58b8950" exitCode=0 Oct 29 21:42:14.681000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.680978 4952 generic.go:356] "Generic (PLEG): container finished" podID="05887799647c9e22c760e149ed245f3e" containerID="08886d7a4243fc68c8749fd96195f4086600663d8bf928299ef03fc8c16de61a" exitCode=0 Oct 29 21:42:14.702386 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.702371 4952 manager.go:324] Recovery completed Oct 29 21:42:14.721941 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.721869 4952 cpu_manager.go:226] "Starting" policy="none" Oct 29 21:42:14.721941 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.721885 4952 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 21:42:14.721941 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.721920 4952 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 21:42:14.722140 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.722027 4952 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 21:42:14.722140 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.722038 4952 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 21:42:14.722140 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.722053 4952 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 21:42:14.722140 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.722059 4952 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 21:42:14.722140 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.722064 4952 policy_none.go:50] "Start" Oct 29 21:42:14.722140 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.722072 4952 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 21:42:14.722140 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.722081 4952 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 21:42:14.734554 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.734539 4952 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 21:42:14.734554 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.734553 4952 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 21:42:14.734634 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.734559 4952 policy_none.go:44] "Start" Oct 29 21:42:14.735816 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:14.735798 4952 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 29 21:42:14.738710 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.738684 4952 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 21:42:14.741297 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.741279 4952 manager.go:341] "Starting Device Plugin manager" Oct 29 21:42:14.741787 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:14.741305 4952 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 21:42:14.741844 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.741790 4952 server.go:87] "Starting device plugin registration server" Oct 29 21:42:14.742053 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.742042 4952 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 21:42:14.742086 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.742053 4952 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 21:42:14.742191 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.742167 4952 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 21:42:14.742399 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.742336 4952 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 21:42:14.742399 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.742392 4952 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 21:42:14.742399 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.742399 4952 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 21:42:14.742585 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.742450 4952 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:42:14.742380897Z","UUID":"4c8aa146-10a0-4fff-aad5-887fd764bdc9","Handler":null,"Name":"","Endpoint":""} Oct 29 21:42:14.743681 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.743663 4952 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 21:42:14.743762 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.743697 4952 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 21:42:14.744025 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:14.744007 4952 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 21:42:14.842726 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.842680 4952 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 21:42:14.844000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.843979 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientMemory" Oct 29 21:42:14.844000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.844008 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 21:42:14.844163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.844019 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-134.ec2.internal" event="NodeHasSufficientPID" Oct 29 21:42:14.844163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.844067 4952 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-134.ec2.internal" Oct 29 21:42:14.883398 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.883364 4952 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-134.ec2.internal" Oct 29 21:42:14.883554 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.883459 4952 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-134.ec2.internal" Oct 29 21:42:14.935973 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.935872 4952 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal"] Oct 29 21:42:14.936609 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.936573 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" event={"ID":"3439c72aff40b3d1f617a398c69856e9","Type":"ContainerStarted","Data":"914f02d6c929974c682b3727af98ff24d228bcbaf94860a4f8bc6f7a386e0285"} Oct 29 21:42:14.936685 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.936617 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" event={"ID":"3439c72aff40b3d1f617a398c69856e9","Type":"ContainerStarted","Data":"f4357f1988c399aa3fb5b008a5f7c483767e42a0414013edf71baa45ca89b904"} Oct 29 21:42:14.936685 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.936661 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" event={"ID":"05887799647c9e22c760e149ed245f3e","Type":"ContainerStarted","Data":"45b2a7d259146657524cc64342addc00a1568c2f85cb3ada92a922e3f10baf90"} Oct 29 21:42:14.936685 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.936674 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" event={"ID":"05887799647c9e22c760e149ed245f3e","Type":"ContainerDied","Data":"08886d7a4243fc68c8749fd96195f4086600663d8bf928299ef03fc8c16de61a"} Oct 29 21:42:14.936798 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.936709 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" event={"ID":"05887799647c9e22c760e149ed245f3e","Type":"ContainerStarted","Data":"44f6a8de86887cbdb030477928904527a0b60058ca463e571005572d449fa744"} Oct 29 21:42:14.936798 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.936726 4952 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:42:14.936875 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.936710 4952 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:42:14.963837 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.963802 4952 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:14.963992 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:14.963867 4952 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:42:14.965135 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:14.965119 4952 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:14.965217 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:14.965158 4952 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.062451 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.062363 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3439c72aff40b3d1f617a398c69856e9-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" (UID: \"3439c72aff40b3d1f617a398c69856e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.062451 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.062411 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.062451 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.062435 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.163312 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.163271 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.163312 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.163316 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.163533 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.163341 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3439c72aff40b3d1f617a398c69856e9-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" (UID: \"3439c72aff40b3d1f617a398c69856e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.163533 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.163370 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.163533 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.163387 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3439c72aff40b3d1f617a398c69856e9-config\") pod \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" (UID: \"3439c72aff40b3d1f617a398c69856e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.163533 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.163430 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05887799647c9e22c760e149ed245f3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" (UID: \"05887799647c9e22c760e149ed245f3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.596873 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.596836 4952 apiserver.go:52] "Watching apiserver" Oct 29 21:42:15.664728 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.664696 4952 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 21:42:15.665427 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.665402 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64","openshift-ingress-canary/ingress-canary-tfwmx","openshift-insights/insights-operator-7c4986d449-mvwvt","openshift-dns/dns-default-mmxmn","openshift-ovn-kubernetes/ovnkube-node-hvpq9","openshift-console-operator/console-operator-7cd56b8dbc-7jz5d","openshift-multus/multus-s8t9b","openshift-network-diagnostics/network-check-target-289sc","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk","openshift-image-registry/node-ca-tjgwl","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5","openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4","openshift-network-console/networking-console-plugin-86c478bb88-zv2wb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n","kube-system/kas-connection-checker-685895c7c9-b425c","openshift-cluster-node-tuning-operator/tuned-f7mnz","openshift-multus/network-metrics-daemon-rcs9l","openshift-network-operator/iptables-alerter-2s6z8","openshift-image-registry/image-registry-748b45bb85-pfdxr","openshift-ingress/router-default-56bd78965c-kscnl","openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6","openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6","kube-system/global-pull-secret-syncer-29g9k","kube-system/konnectivity-agent-krcw7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd","openshift-insights/insights-runtime-extractor-8dsqf","kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal","openshift-multus/multus-additional-cni-plugins-mdxzj","kube-system/kas-connection-checker-685895c7c9-42pjx","openshift-image-registry/image-registry-859d6fc448-k9w4q","openshift-dns/node-resolver-t6k4p","kube-system/kas-connection-checker-685895c7c9-mvw7m"] Oct 29 21:42:15.666636 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.666617 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.667220 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.667199 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:15.669355 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.669334 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:15.670101 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.670086 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.671238 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.671213 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:15.675214 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.675191 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:15.679649 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.679629 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.680119 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.680103 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:15.681066 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.681053 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.682859 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.682839 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.683116 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683099 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 29 21:42:15.683181 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683140 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.683365 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683350 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 29 21:42:15.683494 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683478 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.683552 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683508 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 29 21:42:15.683552 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683518 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 29 21:42:15.683552 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683528 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 29 21:42:15.683552 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683532 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.683881 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683860 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.683881 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683876 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 29 21:42:15.684095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683923 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.684095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683936 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.684095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683969 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.684095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683980 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 29 21:42:15.684095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.683992 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 29 21:42:15.684095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684017 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 29 21:42:15.684095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684037 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.684095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684037 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.684095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684067 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.684095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684038 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.684588 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684189 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.684588 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684518 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.684588 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684528 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 29 21:42:15.684588 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684539 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.684588 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684587 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 29 21:42:15.684804 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.684740 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.689469 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.689452 4952 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.689628 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.689603 4952 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.696834 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.696818 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 29 21:42:15.696834 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.696818 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 29 21:42:15.697012 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.696850 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 29 21:42:15.697012 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.696873 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 29 21:42:15.697012 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.696880 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 21:42:15.697012 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.696893 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.697244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.697036 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 21:42:15.697244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.697084 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.697244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.697090 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 29 21:42:15.697244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.697106 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.697244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.697124 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 21:42:15.697244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.697138 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 29 21:42:15.697244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.697231 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.697473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.697276 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vqm2\"" Oct 29 21:42:15.699769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.699744 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 21:42:15.699883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.699864 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 21:42:15.699999 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.699947 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 21:42:15.699999 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.699874 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 21:42:15.699999 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.699958 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8g25n\"" Oct 29 21:42:15.699999 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.699971 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.700216 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700013 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 29 21:42:15.700216 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700145 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.700326 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700164 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 21:42:15.700326 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700308 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 29 21:42:15.700424 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700349 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 21:42:15.700424 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700367 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zsmh\"" Oct 29 21:42:15.700424 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700411 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.700542 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700422 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 21:42:15.700615 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700599 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.700615 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700609 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 21:42:15.700699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.700609 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 21:42:15.701422 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.701394 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 21:42:15.701609 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.701592 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 21:42:15.701671 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.701648 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.702158 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.702132 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.702619 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.702596 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.702830 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.702811 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.703119 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703104 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.703194 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703160 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.703376 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703359 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 21:42:15.703425 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703412 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.703470 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703439 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.703514 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703482 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 21:42:15.703570 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703562 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.703701 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703684 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.703809 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703721 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.703809 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703784 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 29 21:42:15.703924 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.703848 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.704137 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704114 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cqgbm\"" Oct 29 21:42:15.704221 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704203 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 21:42:15.704322 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704307 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 29 21:42:15.704367 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704320 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lrv4h\"" Oct 29 21:42:15.704434 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704410 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 29 21:42:15.704563 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704547 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bt4z6\"" Oct 29 21:42:15.704662 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704649 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 21:42:15.704774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704748 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rbnwt\"" Oct 29 21:42:15.704821 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704806 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 21:42:15.704874 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704831 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 21:42:15.704983 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.704963 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2qkvr\"" Oct 29 21:42:15.705140 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.705121 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6x2hg\"" Oct 29 21:42:15.705236 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.705214 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cndgh\"" Oct 29 21:42:15.705312 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.705299 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r7b2w\"" Oct 29 21:42:15.705366 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.705130 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gdpmh\"" Oct 29 21:42:15.705366 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.705302 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nll6c\"" Oct 29 21:42:15.705468 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.705428 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 21:42:15.705663 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.705632 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lk29w\"" Oct 29 21:42:15.705839 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.705811 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4df8x\"" Oct 29 21:42:15.705959 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.705874 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jxdl2\"" Oct 29 21:42:15.705959 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.705952 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mldf8\"" Oct 29 21:42:15.706108 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.706064 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 21:42:15.706666 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.706647 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fcqkf\"" Oct 29 21:42:15.710403 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.710389 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 29 21:42:15.710641 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.710627 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 29 21:42:15.713305 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.713290 4952 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 21:42:15.714795 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.714781 4952 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:15.714847 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.714821 4952 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-134.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.717682 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.717667 4952 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 21:42:15.717748 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.717703 4952 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-134.ec2.internal" Oct 29 21:42:15.765672 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.765600 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-log-socket\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.765672 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.765639 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-script-lib\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.765672 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.765662 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv7n9\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-kube-api-access-wv7n9\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.765844 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.765687 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-daemon-config\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.765844 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.765727 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kflk\" (UniqueName: \"kubernetes.io/projected/3dd58b31-fa23-4a8c-b113-ad9479a7b681-kube-api-access-2kflk\") pod \"kas-connection-checker-685895c7c9-b425c\" (UID: \"3dd58b31-fa23-4a8c-b113-ad9479a7b681\") " pod="kube-system/kas-connection-checker-685895c7c9-b425c" Oct 29 21:42:15.765844 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.765782 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-config\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:15.765844 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.765812 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-ovn\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.765844 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.765836 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887lk\" (UniqueName: \"kubernetes.io/projected/059a06d7-5255-47dc-8ce7-07d113e03ca1-kube-api-access-887lk\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.766067 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.765861 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-modprobe-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.766067 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.765958 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nfl\" (UniqueName: \"kubernetes.io/projected/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-kube-api-access-q2nfl\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:15.766139 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766099 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-system-cni-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.766174 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766138 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.766218 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766198 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-systemd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.766268 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766209 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-config\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:15.766268 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766220 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-daemon-config\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.766268 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766220 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-script-lib\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.766268 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766238 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73219cee-7208-4f68-ad0b-bc9cb0bcb199-agent-certs\") pod \"konnectivity-agent-krcw7\" (UID: \"73219cee-7208-4f68-ad0b-bc9cb0bcb199\") " pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:42:15.766418 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766284 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4dbf16f-ae81-461b-8564-99fce7202e3c-crio-socket\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.766418 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766399 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-device-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.766484 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766427 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.766484 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766435 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.766484 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766467 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lzfl\" (UniqueName: \"kubernetes.io/projected/33291806-5f11-4f13-ba59-92a143fe8f0e-kube-api-access-5lzfl\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:15.766577 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766500 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-etc-kubernetes\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.766577 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766523 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61ac615e-a093-42ea-92a4-84d5c22c9f41-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:15.766577 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766534 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73219cee-7208-4f68-ad0b-bc9cb0bcb199-agent-certs\") pod \"konnectivity-agent-krcw7\" (UID: \"73219cee-7208-4f68-ad0b-bc9cb0bcb199\") " pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:42:15.766671 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766618 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.766733 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766714 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.766769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766755 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20070f80-4b65-40ff-9646-e8b7fef85cc1-serving-cert\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:15.766806 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766765 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61ac615e-a093-42ea-92a4-84d5c22c9f41-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:15.766806 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766783 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-run\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.766861 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766809 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-image-registry-private-configuration\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.766899 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766878 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-multus-certs\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.766948 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766893 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20070f80-4b65-40ff-9646-e8b7fef85cc1-serving-cert\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:15.766948 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.766917 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmp9\" (UniqueName: \"kubernetes.io/projected/97611cb7-e267-4d11-b20a-c52d833cdefc-kube-api-access-bhmp9\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.767033 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767019 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-registration-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.767088 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767072 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrjk\" (UniqueName: \"kubernetes.io/projected/20070f80-4b65-40ff-9646-e8b7fef85cc1-kube-api-access-xgrjk\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:15.767134 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767105 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mngv9\" (UniqueName: \"kubernetes.io/projected/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-kube-api-access-mngv9\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:15.767134 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767122 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1461042e-6499-452c-b548-cb243c31b17d-cert\") pod \"ingress-canary-tfwmx\" (UID: \"1461042e-6499-452c-b548-cb243c31b17d\") " pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:15.767230 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767139 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:15.767230 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767147 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-image-registry-private-configuration\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.767230 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767175 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-conf-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.767230 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767192 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61ac615e-a093-42ea-92a4-84d5c22c9f41-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:15.767230 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767209 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg6zw\" (UniqueName: \"kubernetes.io/projected/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-api-access-qg6zw\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.767230 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767225 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-stats-auth\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.767497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767241 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-snapshots\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.767497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767256 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-registry-certificates\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.767497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767278 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-sys\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.767497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767325 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj6l4\" (UniqueName: \"kubernetes.io/projected/61ac615e-a093-42ea-92a4-84d5c22c9f41-kube-api-access-zj6l4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:15.767497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767334 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1461042e-6499-452c-b548-cb243c31b17d-cert\") pod \"ingress-canary-tfwmx\" (UID: \"1461042e-6499-452c-b548-cb243c31b17d\") " pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:15.767497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767436 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-snapshots\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.767497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767439 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-registry-certificates\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.767497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767473 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-bin\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.767497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767493 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-stats-auth\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.767769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767520 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61ac615e-a093-42ea-92a4-84d5c22c9f41-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:15.767769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767552 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.767769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767578 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84058be2-867e-4c5a-a614-bccfd7af96fa-tmp-dir\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:42:15.767769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767619 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-slash\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.767769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767675 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84058be2-867e-4c5a-a614-bccfd7af96fa-tmp-dir\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:42:15.767769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767730 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.767769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767751 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-sys-fs\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.767769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767767 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-tmp\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767786 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-kubernetes\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767835 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767862 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767865 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-tmp\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767870 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-k8s-cni-cncf-io\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767934 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89de847b-bcca-4f80-af6b-071681d3afdb-host\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767961 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-systemd\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.767985 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-lib-modules\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768008 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-var-lib-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768035 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-bound-sa-token\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.768091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768061 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-dbus\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768142 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtcq\" (UniqueName: \"kubernetes.io/projected/4eb95095-a274-43e2-adbd-d88824033bb1-kube-api-access-mgtcq\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768159 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768186 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768204 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-certificates\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768344 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-ca-trust-extracted\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768378 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-var-lib-kubelet\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768394 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-tuned\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768438 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-certificates\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768489 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-hostroot\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768495 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-ca-trust-extracted\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768490 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-tuned\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.768525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768516 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8ph\" (UniqueName: \"kubernetes.io/projected/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-kube-api-access-7x8ph\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768543 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768565 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-socket-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768589 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-netns\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768625 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4dbf16f-ae81-461b-8564-99fce7202e3c-data-volume\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768650 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-netns\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768674 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768695 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4dbf16f-ae81-461b-8564-99fce7202e3c-data-volume\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768701 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768700 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768767 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-host\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768792 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cnibin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768818 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9hc6\" (UniqueName: \"kubernetes.io/projected/587e4eb6-4e54-4c6e-a640-71976f13827c-kube-api-access-r9hc6\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768844 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-service-ca-bundle\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768849 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768928 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9h8r\" (UniqueName: \"kubernetes.io/projected/f04d5d43-128a-4972-b55e-836175e60130-kube-api-access-m9h8r\") pod \"kas-connection-checker-685895c7c9-mvw7m\" (UID: \"f04d5d43-128a-4972-b55e-836175e60130\") " pod="kube-system/kas-connection-checker-685895c7c9-mvw7m" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768947 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768963 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-config\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:15.768970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768979 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.768994 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769015 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-service-ca-bundle\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769024 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmgp\" (UniqueName: \"kubernetes.io/projected/88711e24-d089-4fbb-9b36-fb015fe7f9c9-kube-api-access-bdmgp\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769077 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-system-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769165 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-config\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769179 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-cnibin\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769223 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-serving-cert\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769242 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cni-binary-copy\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769268 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-serving-cert\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769293 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn95n\" (UniqueName: \"kubernetes.io/projected/89de847b-bcca-4f80-af6b-071681d3afdb-kube-api-access-bn95n\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769317 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzw9z\" (UniqueName: \"kubernetes.io/projected/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kube-api-access-rzw9z\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769344 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-kubelet\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769367 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cni-binary-copy\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769372 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-default-certificate\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769388 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-serving-cert\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769399 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjpx\" (UniqueName: \"kubernetes.io/projected/84058be2-867e-4c5a-a614-bccfd7af96fa-kube-api-access-6kjpx\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769428 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr9fw\" (UniqueName: \"kubernetes.io/projected/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-kube-api-access-qr9fw\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769452 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769476 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:15.769568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769485 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-serving-cert\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769596 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysconfig\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769635 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-bound-sa-token\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769601 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-default-certificate\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769691 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-bin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769717 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-kubelet\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769735 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzv72\" (UniqueName: \"kubernetes.io/projected/1b583a61-cc48-4c4e-9e2e-7f6e684d45e8-kube-api-access-qzv72\") pod \"network-check-source-6f78d59f99-zc52n\" (UID: \"1b583a61-cc48-4c4e-9e2e-7f6e684d45e8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769759 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-systemd-units\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769819 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20070f80-4b65-40ff-9646-e8b7fef85cc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769859 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39d65b2-cca6-4f08-bdec-becd6e000c48-tmp-dir\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769884 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsvhn\" (UniqueName: \"kubernetes.io/projected/3398371c-c328-4876-963c-8af098bae440-kube-api-access-rsvhn\") pod \"kas-connection-checker-685895c7c9-42pjx\" (UID: \"3398371c-c328-4876-963c-8af098bae440\") " pod="kube-system/kas-connection-checker-685895c7c9-42pjx" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769931 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/33291806-5f11-4f13-ba59-92a143fe8f0e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769951 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-multus\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.769991 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b39d65b2-cca6-4f08-bdec-becd6e000c48-tmp-dir\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770001 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-trusted-ca\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770019 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770059 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-node-log\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770089 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770122 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91f12b29-bfab-4407-8cae-b618a658c23e-ca-trust-extracted\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770165 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-tmp\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770234 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-kubelet-config\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770234 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/33291806-5f11-4f13-ba59-92a143fe8f0e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770250 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88711e24-d089-4fbb-9b36-fb015fe7f9c9-tmp\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770263 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91f12b29-bfab-4407-8cae-b618a658c23e-ca-trust-extracted\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770277 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770303 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4k74\" (UniqueName: \"kubernetes.io/projected/f8539056-ee01-4262-89b4-1d4b55a5073c-kube-api-access-x4k74\") pod \"volume-data-source-validator-58c9f5f46d-jtzgk\" (UID: \"f8539056-ee01-4262-89b4-1d4b55a5073c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770340 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-installation-pull-secrets\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770390 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb95095-a274-43e2-adbd-d88824033bb1-host-slash\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770417 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-netd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770445 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/587e4eb6-4e54-4c6e-a640-71976f13827c-ovn-node-metrics-cert\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770473 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xdxh\" (UniqueName: \"kubernetes.io/projected/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-kube-api-access-7xdxh\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770542 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-installation-pull-secrets\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770551 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-installation-pull-secrets\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.770644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770585 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-socket-dir-parent\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770682 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770701 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89de847b-bcca-4f80-af6b-071681d3afdb-serviceca\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770717 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e889f05-b993-4423-9b9c-4673ad6e70d3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770758 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/587e4eb6-4e54-4c6e-a640-71976f13827c-ovn-node-metrics-cert\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770768 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-installation-pull-secrets\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770799 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-etc-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770809 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-trusted-ca\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770829 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770854 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-conf\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770919 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89de847b-bcca-4f80-af6b-071681d3afdb-serviceca\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770974 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhssb\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-kube-api-access-qhssb\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.770976 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e889f05-b993-4423-9b9c-4673ad6e70d3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771056 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771084 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771097 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97611cb7-e267-4d11-b20a-c52d833cdefc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771183 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgk9\" (UniqueName: \"kubernetes.io/projected/b39d65b2-cca6-4f08-bdec-becd6e000c48-kube-api-access-kcgk9\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771268 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb5b0077-ddac-46d3-ba06-5fc065be5b53-original-pull-secret\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771342 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84058be2-867e-4c5a-a614-bccfd7af96fa-hosts-file\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771383 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771414 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-image-registry-private-configuration\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771446 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-trusted-ca\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771543 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qnfp\" (UniqueName: \"kubernetes.io/projected/1461042e-6499-452c-b548-cb243c31b17d-kube-api-access-6qnfp\") pod \"ingress-canary-tfwmx\" (UID: \"1461042e-6499-452c-b548-cb243c31b17d\") " pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771589 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-image-registry-private-configuration\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771694 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4eb95095-a274-43e2-adbd-d88824033bb1-iptables-alerter-script\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:42:15.771699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771713 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-config\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771728 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39d65b2-cca6-4f08-bdec-becd6e000c48-config-volume\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771750 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bql\" (UniqueName: \"kubernetes.io/projected/ec6efbe6-a797-4508-a997-c1b3aceec331-kube-api-access-v9bql\") pod \"migrator-66df5d496b-b65z4\" (UID: \"ec6efbe6-a797-4508-a997-c1b3aceec331\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771785 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771814 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-env-overrides\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771839 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73219cee-7208-4f68-ad0b-bc9cb0bcb199-konnectivity-ca\") pod \"konnectivity-agent-krcw7\" (UID: \"73219cee-7208-4f68-ad0b-bc9cb0bcb199\") " pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771862 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-trusted-ca\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771938 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-ovnkube-config\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771950 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4eb95095-a274-43e2-adbd-d88824033bb1-iptables-alerter-script\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771963 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b39d65b2-cca6-4f08-bdec-becd6e000c48-config-volume\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.771991 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/587e4eb6-4e54-4c6e-a640-71976f13827c-env-overrides\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.772500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.772068 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73219cee-7208-4f68-ad0b-bc9cb0bcb199-konnectivity-ca\") pod \"konnectivity-agent-krcw7\" (UID: \"73219cee-7208-4f68-ad0b-bc9cb0bcb199\") " pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:42:15.789347 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.789312 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kflk\" (UniqueName: \"kubernetes.io/projected/3dd58b31-fa23-4a8c-b113-ad9479a7b681-kube-api-access-2kflk\") pod \"kas-connection-checker-685895c7c9-b425c\" (UID: \"3dd58b31-fa23-4a8c-b113-ad9479a7b681\") " pod="kube-system/kas-connection-checker-685895c7c9-b425c" Oct 29 21:42:15.789457 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.789312 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nfl\" (UniqueName: \"kubernetes.io/projected/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-kube-api-access-q2nfl\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:15.789457 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.789384 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv7n9\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-kube-api-access-wv7n9\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.790043 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.790019 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj6l4\" (UniqueName: \"kubernetes.io/projected/61ac615e-a093-42ea-92a4-84d5c22c9f41-kube-api-access-zj6l4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5\" (UID: \"61ac615e-a093-42ea-92a4-84d5c22c9f41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" Oct 29 21:42:15.790114 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.790051 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrjk\" (UniqueName: \"kubernetes.io/projected/20070f80-4b65-40ff-9646-e8b7fef85cc1-kube-api-access-xgrjk\") pod \"console-operator-7cd56b8dbc-7jz5d\" (UID: \"20070f80-4b65-40ff-9646-e8b7fef85cc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:15.790378 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.790361 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-bound-sa-token\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.790452 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.790373 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngv9\" (UniqueName: \"kubernetes.io/projected/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-kube-api-access-mngv9\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:15.790550 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.790533 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtcq\" (UniqueName: \"kubernetes.io/projected/4eb95095-a274-43e2-adbd-d88824033bb1-kube-api-access-mgtcq\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:42:15.790830 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.790811 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmp9\" (UniqueName: \"kubernetes.io/projected/97611cb7-e267-4d11-b20a-c52d833cdefc-kube-api-access-bhmp9\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.793857 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.793837 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg6zw\" (UniqueName: \"kubernetes.io/projected/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-api-access-qg6zw\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.793959 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.793942 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887lk\" (UniqueName: \"kubernetes.io/projected/059a06d7-5255-47dc-8ce7-07d113e03ca1-kube-api-access-887lk\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.794009 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.793981 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8ph\" (UniqueName: \"kubernetes.io/projected/cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9-kube-api-access-7x8ph\") pod \"service-ca-operator-544d749c8c-h4zk6\" (UID: \"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" Oct 29 21:42:15.794113 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.794098 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zknj5\" (UniqueName: \"kubernetes.io/projected/3dbda061-df54-482d-96c6-851f1acf58a4-kube-api-access-zknj5\") pod \"network-check-target-289sc\" (UID: \"3dbda061-df54-482d-96c6-851f1acf58a4\") " pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:15.796900 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.796877 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9h8r\" (UniqueName: \"kubernetes.io/projected/f04d5d43-128a-4972-b55e-836175e60130-kube-api-access-m9h8r\") pod \"kas-connection-checker-685895c7c9-mvw7m\" (UID: \"f04d5d43-128a-4972-b55e-836175e60130\") " pod="kube-system/kas-connection-checker-685895c7c9-mvw7m" Oct 29 21:42:15.797374 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.797323 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsvhn\" (UniqueName: \"kubernetes.io/projected/3398371c-c328-4876-963c-8af098bae440-kube-api-access-rsvhn\") pod \"kas-connection-checker-685895c7c9-42pjx\" (UID: \"3398371c-c328-4876-963c-8af098bae440\") " pod="kube-system/kas-connection-checker-685895c7c9-42pjx" Oct 29 21:42:15.797459 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.797393 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn95n\" (UniqueName: \"kubernetes.io/projected/89de847b-bcca-4f80-af6b-071681d3afdb-kube-api-access-bn95n\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:42:15.797534 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.797518 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjpx\" (UniqueName: \"kubernetes.io/projected/84058be2-867e-4c5a-a614-bccfd7af96fa-kube-api-access-6kjpx\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:42:15.797672 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.797653 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-bound-sa-token\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.797718 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.797690 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9hc6\" (UniqueName: \"kubernetes.io/projected/587e4eb6-4e54-4c6e-a640-71976f13827c-kube-api-access-r9hc6\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.797749 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.797712 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4k74\" (UniqueName: \"kubernetes.io/projected/f8539056-ee01-4262-89b4-1d4b55a5073c-kube-api-access-x4k74\") pod \"volume-data-source-validator-58c9f5f46d-jtzgk\" (UID: \"f8539056-ee01-4262-89b4-1d4b55a5073c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jtzgk" Oct 29 21:42:15.797795 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.797780 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr9fw\" (UniqueName: \"kubernetes.io/projected/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-kube-api-access-qr9fw\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.797831 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.797809 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhssb\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-kube-api-access-qhssb\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.798782 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.798753 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgk9\" (UniqueName: \"kubernetes.io/projected/b39d65b2-cca6-4f08-bdec-becd6e000c48-kube-api-access-kcgk9\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:15.798856 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.798804 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bql\" (UniqueName: \"kubernetes.io/projected/ec6efbe6-a797-4508-a997-c1b3aceec331-kube-api-access-v9bql\") pod \"migrator-66df5d496b-b65z4\" (UID: \"ec6efbe6-a797-4508-a997-c1b3aceec331\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b65z4" Oct 29 21:42:15.799016 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.799000 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xdxh\" (UniqueName: \"kubernetes.io/projected/1f4b9c11-19cd-46b9-8096-5c9375fa7e91-kube-api-access-7xdxh\") pod \"insights-operator-7c4986d449-mvwvt\" (UID: \"1f4b9c11-19cd-46b9-8096-5c9375fa7e91\") " pod="openshift-insights/insights-operator-7c4986d449-mvwvt" Oct 29 21:42:15.799063 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.799039 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lzfl\" (UniqueName: \"kubernetes.io/projected/33291806-5f11-4f13-ba59-92a143fe8f0e-kube-api-access-5lzfl\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:15.799232 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.799217 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzw9z\" (UniqueName: \"kubernetes.io/projected/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kube-api-access-rzw9z\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.802376 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.802359 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzv72\" (UniqueName: \"kubernetes.io/projected/1b583a61-cc48-4c4e-9e2e-7f6e684d45e8-kube-api-access-qzv72\") pod \"network-check-source-6f78d59f99-zc52n\" (UID: \"1b583a61-cc48-4c4e-9e2e-7f6e684d45e8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zc52n" Oct 29 21:42:15.805269 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.805254 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmgp\" (UniqueName: \"kubernetes.io/projected/88711e24-d089-4fbb-9b36-fb015fe7f9c9-kube-api-access-bdmgp\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.808452 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.808435 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qnfp\" (UniqueName: \"kubernetes.io/projected/1461042e-6499-452c-b548-cb243c31b17d-kube-api-access-6qnfp\") pod \"ingress-canary-tfwmx\" (UID: \"1461042e-6499-452c-b548-cb243c31b17d\") " pod="openshift-ingress-canary/ingress-canary-tfwmx" Oct 29 21:42:15.872347 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872317 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.872347 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872349 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysconfig\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.872511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872365 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-bin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.872511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872379 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-kubelet\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.872511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872420 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-kubelet\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.872511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872444 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-bin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.872511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872460 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.872511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872444 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysconfig\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.872511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872470 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-systemd-units\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.872511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872507 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-systemd-units\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872537 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-multus\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872561 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-var-lib-cni-multus\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872570 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872596 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-node-log\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872597 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872611 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872625 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-node-log\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872627 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-kubelet-config\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872645 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872655 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872663 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb95095-a274-43e2-adbd-d88824033bb1-host-slash\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872689 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-kubelet-config\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872700 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-netd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872719 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb95095-a274-43e2-adbd-d88824033bb1-host-slash\") pod \"iptables-alerter-2s6z8\" (UID: \"4eb95095-a274-43e2-adbd-d88824033bb1\") " pod="openshift-network-operator/iptables-alerter-2s6z8" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872741 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-socket-dir-parent\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872760 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-etc-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.872770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872774 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.872733 4952 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872829 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-netd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872853 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-etc-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872858 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-socket-dir-parent\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.872878 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.872894 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748b45bb85-pfdxr: secret "image-registry-tls" not found Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872791 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-conf\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.872962 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-conf\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.872879 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert podName:1e889f05-b993-4423-9b9c-4673ad6e70d3 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:16.372862 +0000 UTC m=+1.847450413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zv2wb" (UID: "1e889f05-b993-4423-9b9c-4673ad6e70d3") : secret "networking-console-plugin-cert" not found Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873005 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls podName:5ca158db-9c67-4fb5-bc6b-7c15ceee8772 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:16.372988097 +0000 UTC m=+1.847576497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls") pod "image-registry-748b45bb85-pfdxr" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772") : secret "image-registry-tls" not found Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873029 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873047 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84058be2-867e-4c5a-a614-bccfd7af96fa-hosts-file\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873060 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873081 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873086 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873101 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-log-socket\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873129 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-ovn\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873135 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84058be2-867e-4c5a-a614-bccfd7af96fa-hosts-file\") pod \"node-resolver-t6k4p\" (UID: \"84058be2-867e-4c5a-a614-bccfd7af96fa\") " pod="openshift-dns/node-resolver-t6k4p" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873131 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-log-socket\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873153 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873153 4952 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873153 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-modprobe-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873165 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-ovn\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873190 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-system-cni-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873209 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-system-cni-dir\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873241 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-modprobe-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873236 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-systemd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873266 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls podName:33291806-5f11-4f13-ba59-92a143fe8f0e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:16.373250513 +0000 UTC m=+1.847838925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-m4ks6" (UID: "33291806-5f11-4f13-ba59-92a143fe8f0e") : secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873277 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-systemd\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873306 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4dbf16f-ae81-461b-8564-99fce7202e3c-crio-socket\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.873310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873322 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-device-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873339 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873353 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-etc-kubernetes\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873372 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873389 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873402 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-device-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873422 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4dbf16f-ae81-461b-8564-99fce7202e3c-crio-socket\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873425 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-etc-kubernetes\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873442 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-run\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873457 4952 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873465 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-multus-certs\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873469 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873480 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859d6fc448-k9w4q: secret "image-registry-tls" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873482 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-registration-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873500 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:16.373487718 +0000 UTC m=+1.848076137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : secret "router-metrics-certs-default" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873527 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls podName:91f12b29-bfab-4407-8cae-b618a658c23e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:16.373515285 +0000 UTC m=+1.848103693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls") pod "image-registry-859d6fc448-k9w4q" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e") : secret "image-registry-tls" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873528 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-registration-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873547 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873548 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-multus-certs\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873526 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-run\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873599 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-conf-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873621 4952 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873660 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-sys\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873676 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:16.373663957 +0000 UTC m=+1.848252371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : secret "metrics-daemon-secret" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873674 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-multus-conf-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873691 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-sys\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873702 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-bin\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873728 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-slash\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873757 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873782 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-sys-fs\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873797 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-cni-bin\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873807 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-kubernetes\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873832 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873798 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-slash\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873836 4952 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873836 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-kubernetes\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873873 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-k8s-cni-cncf-io\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.873884 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls podName:a4dbf16f-ae81-461b-8564-99fce7202e3c nodeName:}" failed. No retries permitted until 2026-10-29 21:42:16.373872913 +0000 UTC m=+1.848461314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8dsqf" (UID: "a4dbf16f-ae81-461b-8564-99fce7202e3c") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873895 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-k8s-cni-cncf-io\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873834 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-sys-fs\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873935 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89de847b-bcca-4f80-af6b-071681d3afdb-host\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873954 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-sysctl-d\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873975 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89de847b-bcca-4f80-af6b-071681d3afdb-host\") pod \"node-ca-tjgwl\" (UID: \"89de847b-bcca-4f80-af6b-071681d3afdb\") " pod="openshift-image-registry/node-ca-tjgwl" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873976 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-systemd\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.873994 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4dbf16f-ae81-461b-8564-99fce7202e3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874009 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-lib-modules\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874031 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-var-lib-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874042 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-etc-systemd\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874046 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-dbus\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874073 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-var-lib-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874080 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874085 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb5b0077-ddac-46d3-ba06-5fc065be5b53-dbus\") pod \"global-pull-secret-syncer-29g9k\" (UID: \"fb5b0077-ddac-46d3-ba06-5fc065be5b53\") " pod="kube-system/global-pull-secret-syncer-29g9k" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874093 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-lib-modules\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874108 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-var-lib-kubelet\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874133 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-hostroot\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874157 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-socket-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874169 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-run-openvswitch\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874177 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-var-lib-kubelet\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874180 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-netns\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874192 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-hostroot\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874210 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-netns\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874236 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874212 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcf1d737-3cce-4759-8bde-21c7e0dff0ae-socket-dir\") pod \"aws-ebs-csi-driver-node-vq7qd\" (UID: \"dcf1d737-3cce-4759-8bde-21c7e0dff0ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vq7qd" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874259 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-host\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874211 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-host-run-netns\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874283 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cnibin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874284 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-run-netns\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874317 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88711e24-d089-4fbb-9b36-fb015fe7f9c9-host\") pod \"tuned-f7mnz\" (UID: \"88711e24-d089-4fbb-9b36-fb015fe7f9c9\") " pod="openshift-cluster-node-tuning-operator/tuned-f7mnz" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874324 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-cnibin\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874317 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.874339 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:16.374321761 +0000 UTC m=+1.848910187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : configmap references non-existent config key: service-ca.crt Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874367 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.874389 4952 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874412 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874386 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.874438 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls podName:b39d65b2-cca6-4f08-bdec-becd6e000c48 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:16.374425282 +0000 UTC m=+1.849013678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls") pod "dns-default-mmxmn" (UID: "b39d65b2-cca6-4f08-bdec-becd6e000c48") : secret "dns-default-metrics-tls" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.874460 4952 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874471 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-system-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.874428 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:15.874488 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls podName:53ff731b-5c02-4366-b6b2-c5ecfdc8be38 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:16.374479384 +0000 UTC m=+1.849067822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-46z64" (UID: "53ff731b-5c02-4366-b6b2-c5ecfdc8be38") : secret "samples-operator-tls" not found Oct 29 21:42:15.876542 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874516 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-cnibin\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.876542 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874537 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-kubelet\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.876542 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874592 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97611cb7-e267-4d11-b20a-c52d833cdefc-cnibin\") pod \"multus-additional-cni-plugins-mdxzj\" (UID: \"97611cb7-e267-4d11-b20a-c52d833cdefc\") " pod="openshift-multus/multus-additional-cni-plugins-mdxzj" Oct 29 21:42:15.876542 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874516 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a8cd583-ba4f-4f3a-9f42-f43d63462d32-system-cni-dir\") pod \"multus-s8t9b\" (UID: \"1a8cd583-ba4f-4f3a-9f42-f43d63462d32\") " pod="openshift-multus/multus-s8t9b" Oct 29 21:42:15.876542 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.874612 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/587e4eb6-4e54-4c6e-a640-71976f13827c-host-kubelet\") pod \"ovnkube-node-hvpq9\" (UID: \"587e4eb6-4e54-4c6e-a640-71976f13827c\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:15.957458 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.957423 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:15.967627 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:15.967605 4952 scope.go:122] "RemoveContainer" containerID="b66b5d7d234cfd8206a074cdeb2ee9e4cc101586f990e55edee853f3b12fd063" Oct 29 21:42:16.055289 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.055261 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:16.379628 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.379579 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:16.379628 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.379628 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.379663 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.379686 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.379710 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.379735 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379739 4952 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.379754 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379798 4952 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.379811 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379838 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379798 4952 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379860 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748b45bb85-pfdxr: secret "image-registry-tls" not found Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379862 4952 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379879 4952 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379890 4952 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 21:42:16.379892 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379849 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls podName:53ff731b-5c02-4366-b6b2-c5ecfdc8be38 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:17.379829508 +0000 UTC m=+2.854417918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-46z64" (UID: "53ff731b-5c02-4366-b6b2-c5ecfdc8be38") : secret "samples-operator-tls" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379882 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.379944 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379957 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:17.379942302 +0000 UTC m=+2.854530717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : secret "metrics-daemon-secret" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379957 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859d6fc448-k9w4q: secret "image-registry-tls" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.379994 4952 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.380023 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls podName:33291806-5f11-4f13-ba59-92a143fe8f0e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:17.379993352 +0000 UTC m=+2.854581747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-m4ks6" (UID: "33291806-5f11-4f13-ba59-92a143fe8f0e") : secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.380047 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert podName:1e889f05-b993-4423-9b9c-4673ad6e70d3 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:17.380037658 +0000 UTC m=+2.854626063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zv2wb" (UID: "1e889f05-b993-4423-9b9c-4673ad6e70d3") : secret "networking-console-plugin-cert" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.380077 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls podName:5ca158db-9c67-4fb5-bc6b-7c15ceee8772 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:17.38005947 +0000 UTC m=+2.854647881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls") pod "image-registry-748b45bb85-pfdxr" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772") : secret "image-registry-tls" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.380100 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:17.380089716 +0000 UTC m=+2.854678116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : secret "router-metrics-certs-default" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.380120 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls podName:b39d65b2-cca6-4f08-bdec-becd6e000c48 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:17.380110971 +0000 UTC m=+2.854699371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls") pod "dns-default-mmxmn" (UID: "b39d65b2-cca6-4f08-bdec-becd6e000c48") : secret "dns-default-metrics-tls" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.380140 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls podName:91f12b29-bfab-4407-8cae-b618a658c23e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:17.380132026 +0000 UTC m=+2.854720425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls") pod "image-registry-859d6fc448-k9w4q" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e") : secret "image-registry-tls" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.380163 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls podName:a4dbf16f-ae81-461b-8564-99fce7202e3c nodeName:}" failed. No retries permitted until 2026-10-29 21:42:17.380153926 +0000 UTC m=+2.854742329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8dsqf" (UID: "a4dbf16f-ae81-461b-8564-99fce7202e3c") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.380217 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:16.380500 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.380335 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:17.380324081 +0000 UTC m=+2.854912478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : configmap references non-existent config key: service-ca.crt Oct 29 21:42:16.694118 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.694040 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 21:42:16.694494 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.694482 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/1.log" Oct 29 21:42:16.694537 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.694513 4952 generic.go:356] "Generic (PLEG): container finished" podID="20070f80-4b65-40ff-9646-e8b7fef85cc1" containerID="f5da38fd3443febad62ffd6fbe836a07bb27d50ec1618bd0d73ae93c414856ad" exitCode=255 Oct 29 21:42:16.694636 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.694610 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" event={"ID":"20070f80-4b65-40ff-9646-e8b7fef85cc1","Type":"ContainerDied","Data":"f5da38fd3443febad62ffd6fbe836a07bb27d50ec1618bd0d73ae93c414856ad"} Oct 29 21:42:16.694691 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.694668 4952 scope.go:122] "RemoveContainer" containerID="b66b5d7d234cfd8206a074cdeb2ee9e4cc101586f990e55edee853f3b12fd063" Oct 29 21:42:16.694921 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:16.694875 4952 scope.go:122] "RemoveContainer" containerID="f5da38fd3443febad62ffd6fbe836a07bb27d50ec1618bd0d73ae93c414856ad" Oct 29 21:42:16.695125 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:16.695096 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7jz5d_openshift-console-operator(20070f80-4b65-40ff-9646-e8b7fef85cc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" podUID="20070f80-4b65-40ff-9646-e8b7fef85cc1" Oct 29 21:42:17.182016 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.181976 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:17.328403 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.328362 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:17.387345 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.387311 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:17.387345 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.387353 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.387373 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.387421 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.387455 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387488 4952 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387498 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.387478868 +0000 UTC m=+4.862067282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : configmap references non-existent config key: service-ca.crt Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387531 4952 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387542 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls podName:b39d65b2-cca6-4f08-bdec-becd6e000c48 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.387526242 +0000 UTC m=+4.862114710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls") pod "dns-default-mmxmn" (UID: "b39d65b2-cca6-4f08-bdec-becd6e000c48") : secret "dns-default-metrics-tls" not found Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387565 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert podName:1e889f05-b993-4423-9b9c-4673ad6e70d3 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.387555687 +0000 UTC m=+4.862144097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zv2wb" (UID: "1e889f05-b993-4423-9b9c-4673ad6e70d3") : secret "networking-console-plugin-cert" not found Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.387529 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387578 4952 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:17.387582 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387580 4952 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.387619 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387641 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls podName:53ff731b-5c02-4366-b6b2-c5ecfdc8be38 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.387625959 +0000 UTC m=+4.862214362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-46z64" (UID: "53ff731b-5c02-4366-b6b2-c5ecfdc8be38") : secret "samples-operator-tls" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387648 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387666 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748b45bb85-pfdxr: secret "image-registry-tls" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387671 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls podName:33291806-5f11-4f13-ba59-92a143fe8f0e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.38766222 +0000 UTC m=+4.862250619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-m4ks6" (UID: "33291806-5f11-4f13-ba59-92a143fe8f0e") : secret "cluster-monitoring-operator-tls" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387685 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387699 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859d6fc448-k9w4q: secret "image-registry-tls" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.387703 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.387729 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387744 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls podName:91f12b29-bfab-4407-8cae-b618a658c23e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.387734851 +0000 UTC m=+4.862323250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls") pod "image-registry-859d6fc448-k9w4q" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e") : secret "image-registry-tls" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387750 4952 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387772 4952 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387781 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls podName:5ca158db-9c67-4fb5-bc6b-7c15ceee8772 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.387766368 +0000 UTC m=+4.862354763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls") pod "image-registry-748b45bb85-pfdxr" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772") : secret "image-registry-tls" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387803 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.387796545 +0000 UTC m=+4.862384944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : secret "metrics-daemon-secret" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.387799 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387813 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.387807419 +0000 UTC m=+4.862395815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : secret "router-metrics-certs-default" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387837 4952 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:17.388436 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.387865 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls podName:a4dbf16f-ae81-461b-8564-99fce7202e3c nodeName:}" failed. No retries permitted until 2026-10-29 21:42:19.387857579 +0000 UTC m=+4.862445978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8dsqf" (UID: "a4dbf16f-ae81-461b-8564-99fce7202e3c") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:17.539811 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.539730 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:17.668109 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.668081 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:17.700143 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.700118 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 21:42:17.700528 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.700429 4952 scope.go:122] "RemoveContainer" containerID="f5da38fd3443febad62ffd6fbe836a07bb27d50ec1618bd0d73ae93c414856ad" Oct 29 21:42:17.700618 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:17.700600 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7jz5d_openshift-console-operator(20070f80-4b65-40ff-9646-e8b7fef85cc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" podUID="20070f80-4b65-40ff-9646-e8b7fef85cc1" Oct 29 21:42:17.839168 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:17.839125 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:18.250406 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:18.250315 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:18.522474 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:18.522403 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t6k4p_84058be2-867e-4c5a-a614-bccfd7af96fa/dns-node-resolver/0.log" Oct 29 21:42:19.064503 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.064457 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:19.259397 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.259363 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:19.322624 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.322553 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tjgwl_89de847b-bcca-4f80-af6b-071681d3afdb/node-ca/0.log" Oct 29 21:42:19.410105 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410065 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:19.410288 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410117 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:19.410288 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410156 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:19.410288 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410190 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:19.410288 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410216 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:19.410288 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410219 4952 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 21:42:19.410288 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410247 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:19.410288 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410276 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:19.410288 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410292 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert podName:1e889f05-b993-4423-9b9c-4673ad6e70d3 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.410274355 +0000 UTC m=+8.884862767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zv2wb" (UID: "1e889f05-b993-4423-9b9c-4673ad6e70d3") : secret "networking-console-plugin-cert" not found Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410278 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410294 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748b45bb85-pfdxr: secret "image-registry-tls" not found Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410399 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410307 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410446 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859d6fc448-k9w4q: secret "image-registry-tls" not found Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410339 4952 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410354 4952 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410377 4952 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410433 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls podName:5ca158db-9c67-4fb5-bc6b-7c15ceee8772 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.410418527 +0000 UTC m=+8.885006942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls") pod "image-registry-748b45bb85-pfdxr" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772") : secret "image-registry-tls" not found Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410539 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410563 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.410584 4952 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410591 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.410578544 +0000 UTC m=+8.885166945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : configmap references non-existent config key: service-ca.crt Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410610 4952 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 21:42:19.410650 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410636 4952 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 21:42:19.411122 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410671 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.410656992 +0000 UTC m=+8.885245403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : secret "router-metrics-certs-default" not found Oct 29 21:42:19.411122 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410738 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls podName:91f12b29-bfab-4407-8cae-b618a658c23e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.410721326 +0000 UTC m=+8.885309722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls") pod "image-registry-859d6fc448-k9w4q" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e") : secret "image-registry-tls" not found Oct 29 21:42:19.411122 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410759 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls podName:a4dbf16f-ae81-461b-8564-99fce7202e3c nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.410750128 +0000 UTC m=+8.885338523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8dsqf" (UID: "a4dbf16f-ae81-461b-8564-99fce7202e3c") : secret "insights-runtime-extractor-tls" not found Oct 29 21:42:19.411122 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410771 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs podName:e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.410765997 +0000 UTC m=+8.885354393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs") pod "network-metrics-daemon-rcs9l" (UID: "e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8") : secret "metrics-daemon-secret" not found Oct 29 21:42:19.411122 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410780 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls podName:b39d65b2-cca6-4f08-bdec-becd6e000c48 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.410775714 +0000 UTC m=+8.885364110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls") pod "dns-default-mmxmn" (UID: "b39d65b2-cca6-4f08-bdec-becd6e000c48") : secret "dns-default-metrics-tls" not found Oct 29 21:42:19.411122 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:19.410789 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls podName:53ff731b-5c02-4366-b6b2-c5ecfdc8be38 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:23.41078466 +0000 UTC m=+8.885373056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-46z64" (UID: "53ff731b-5c02-4366-b6b2-c5ecfdc8be38") : secret "samples-operator-tls" not found Oct 29 21:42:19.412726 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.412707 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/33291806-5f11-4f13-ba59-92a143fe8f0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-m4ks6\" (UID: \"33291806-5f11-4f13-ba59-92a143fe8f0e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:19.567565 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.567531 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" Oct 29 21:42:19.629809 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.629717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:19.848078 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.847992 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:19.863711 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.863685 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6"] Oct 29 21:42:19.866676 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:19.866651 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33291806_5f11_4f13_ba59_92a143fe8f0e.slice/crio-8759cbe672b81a3fcd51b3161750b281783648e9ba755ff264323cd7cabc52fe WatchSource:0}: Error finding container 8759cbe672b81a3fcd51b3161750b281783648e9ba755ff264323cd7cabc52fe: Status 404 returned error can't find the container with id 8759cbe672b81a3fcd51b3161750b281783648e9ba755ff264323cd7cabc52fe Oct 29 21:42:19.868588 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:19.868574 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:42:20.126603 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:20.126521 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tfwmx_1461042e-6499-452c-b548-cb243c31b17d/serve-healthcheck-canary/0.log" Oct 29 21:42:20.431479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:20.431396 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:20.524029 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:20.523999 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-b65z4_ec6efbe6-a797-4508-a997-c1b3aceec331/migrator/0.log" Oct 29 21:42:20.638566 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:20.638470 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:20.710760 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:20.710682 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" event={"ID":"33291806-5f11-4f13-ba59-92a143fe8f0e","Type":"ContainerStarted","Data":"8759cbe672b81a3fcd51b3161750b281783648e9ba755ff264323cd7cabc52fe"} Oct 29 21:42:20.925108 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:20.925074 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5_61ac615e-a093-42ea-92a4-84d5c22c9f41/kube-storage-version-migrator-operator/0.log" Oct 29 21:42:21.028287 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:21.028210 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:21.284640 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:21.284614 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:21.640799 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:21.640753 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:21.715166 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:21.715112 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-m4ks6" event={"ID":"33291806-5f11-4f13-ba59-92a143fe8f0e","Type":"ContainerStarted","Data":"d582439d1073929e31b86d4d2c90243c02f566837f43fcf46d8964aa0d117392"} Oct 29 21:42:21.836552 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:21.836507 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:22.014514 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:22.014432 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:22.130555 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:22.130523 4952 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:22.130963 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:22.130946 4952 scope.go:122] "RemoveContainer" containerID="f5da38fd3443febad62ffd6fbe836a07bb27d50ec1618bd0d73ae93c414856ad" Oct 29 21:42:22.131174 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:22.131152 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7jz5d_openshift-console-operator(20070f80-4b65-40ff-9646-e8b7fef85cc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" podUID="20070f80-4b65-40ff-9646-e8b7fef85cc1" Oct 29 21:42:22.233585 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:22.233559 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:22.410629 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:22.410597 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:23.043386 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.043356 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:23.439091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.439062 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:23.448057 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.448027 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:23.448177 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.448077 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:23.448177 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.448119 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:23.448177 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.448137 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:23.448177 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.448159 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:23.448163 4952 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.448194 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.448234 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:23.448199 4952 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:23.448262 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert podName:1e889f05-b993-4423-9b9c-4673ad6e70d3 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:31.448244351 +0000 UTC m=+16.922832768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zv2wb" (UID: "1e889f05-b993-4423-9b9c-4673ad6e70d3") : secret "networking-console-plugin-cert" not found Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:23.448284 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs podName:059a06d7-5255-47dc-8ce7-07d113e03ca1 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:31.448274011 +0000 UTC m=+16.922862409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs") pod "router-default-56bd78965c-kscnl" (UID: "059a06d7-5255-47dc-8ce7-07d113e03ca1") : secret "router-metrics-certs-default" not found Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:23.448300 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.448306 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:23.448311 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748b45bb85-pfdxr: secret "image-registry-tls" not found Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:23.448405 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls podName:5ca158db-9c67-4fb5-bc6b-7c15ceee8772 nodeName:}" failed. No retries permitted until 2026-10-29 21:42:31.448383777 +0000 UTC m=+16.922972188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls") pod "image-registry-748b45bb85-pfdxr" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772") : secret "image-registry-tls" not found Oct 29 21:42:23.448447 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.448331 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:23.449005 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:23.448244 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 21:42:23.449005 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:23.448489 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859d6fc448-k9w4q: secret "image-registry-tls" not found Oct 29 21:42:23.449005 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:23.448525 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls podName:91f12b29-bfab-4407-8cae-b618a658c23e nodeName:}" failed. No retries permitted until 2026-10-29 21:42:31.448513766 +0000 UTC m=+16.923102178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls") pod "image-registry-859d6fc448-k9w4q" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e") : secret "image-registry-tls" not found Oct 29 21:42:23.449215 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.449194 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/059a06d7-5255-47dc-8ce7-07d113e03ca1-service-ca-bundle\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:23.450523 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.450496 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8-metrics-certs\") pod \"network-metrics-daemon-rcs9l\" (UID: \"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8\") " pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:23.450814 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.450790 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ff731b-5c02-4366-b6b2-c5ecfdc8be38-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-46z64\" (UID: \"53ff731b-5c02-4366-b6b2-c5ecfdc8be38\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:23.450924 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.450861 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4dbf16f-ae81-461b-8564-99fce7202e3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8dsqf\" (UID: \"a4dbf16f-ae81-461b-8564-99fce7202e3c\") " pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:23.450992 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.450933 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b39d65b2-cca6-4f08-bdec-becd6e000c48-metrics-tls\") pod \"dns-default-mmxmn\" (UID: \"b39d65b2-cca6-4f08-bdec-becd6e000c48\") " pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:23.470353 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.470331 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" Oct 29 21:42:23.475669 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.475650 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcs9l" Oct 29 21:42:23.480433 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.480413 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:23.481523 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.481504 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8dsqf" Oct 29 21:42:23.636781 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.636747 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:23.787394 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.787322 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64"] Oct 29 21:42:23.794180 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.794154 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rcs9l"] Oct 29 21:42:23.799896 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.799875 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mmxmn"] Oct 29 21:42:23.804244 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:23.804219 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69b8cc6_b91f_4cf6_9b3c_92324e5be0e8.slice/crio-050522d58115f4eea600d6b77ed7db4ccac9b25a49b3bcad2e6a9257a4c77790 WatchSource:0}: Error finding container 050522d58115f4eea600d6b77ed7db4ccac9b25a49b3bcad2e6a9257a4c77790: Status 404 returned error can't find the container with id 050522d58115f4eea600d6b77ed7db4ccac9b25a49b3bcad2e6a9257a4c77790 Oct 29 21:42:23.804454 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:23.804434 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39d65b2_cca6_4f08_bdec_becd6e000c48.slice/crio-6739c23a6099d2d0e04b83faf106284b1400b572cc7950100d3771630f1133f4 WatchSource:0}: Error finding container 6739c23a6099d2d0e04b83faf106284b1400b572cc7950100d3771630f1133f4: Status 404 returned error can't find the container with id 6739c23a6099d2d0e04b83faf106284b1400b572cc7950100d3771630f1133f4 Oct 29 21:42:23.808087 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.807993 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8dsqf"] Oct 29 21:42:23.810824 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:23.810801 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4dbf16f_ae81_461b_8564_99fce7202e3c.slice/crio-6938e61711525942ee58807dd7bf8a32494117e7df3f85c301b22f5d3138ddfa WatchSource:0}: Error finding container 6938e61711525942ee58807dd7bf8a32494117e7df3f85c301b22f5d3138ddfa: Status 404 returned error can't find the container with id 6938e61711525942ee58807dd7bf8a32494117e7df3f85c301b22f5d3138ddfa Oct 29 21:42:23.836718 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:23.836689 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:24.256886 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:24.256849 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:24.711742 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:24.711576 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:24.732595 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:24.732372 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8dsqf" event={"ID":"a4dbf16f-ae81-461b-8564-99fce7202e3c","Type":"ContainerStarted","Data":"8f0de5683ff3a42cdfcbe563e52c40e1b33ee7a975440af44baf8f81b58767a4"} Oct 29 21:42:24.732595 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:24.732513 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8dsqf" event={"ID":"a4dbf16f-ae81-461b-8564-99fce7202e3c","Type":"ContainerStarted","Data":"c65644e7351c26ac6e22ee8cd0911cea8f809145447a23b86f5f707b84f7e955"} Oct 29 21:42:24.732595 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:24.732544 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8dsqf" event={"ID":"a4dbf16f-ae81-461b-8564-99fce7202e3c","Type":"ContainerStarted","Data":"6938e61711525942ee58807dd7bf8a32494117e7df3f85c301b22f5d3138ddfa"} Oct 29 21:42:24.733855 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:24.733792 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" event={"ID":"53ff731b-5c02-4366-b6b2-c5ecfdc8be38","Type":"ContainerStarted","Data":"32b44e4e373be518c94fed5ec1405b3ea03ad8ec82c46d1266019a0d0da00d61"} Oct 29 21:42:24.735311 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:24.735281 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rcs9l" event={"ID":"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8","Type":"ContainerStarted","Data":"050522d58115f4eea600d6b77ed7db4ccac9b25a49b3bcad2e6a9257a4c77790"} Oct 29 21:42:24.736664 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:24.736642 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmxmn" event={"ID":"b39d65b2-cca6-4f08-bdec-becd6e000c48","Type":"ContainerStarted","Data":"6739c23a6099d2d0e04b83faf106284b1400b572cc7950100d3771630f1133f4"} Oct 29 21:42:25.205991 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:25.205950 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:25.395275 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:25.395240 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:25.573343 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:25.573302 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:25.743564 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:25.743490 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rcs9l" event={"ID":"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8","Type":"ContainerStarted","Data":"43d479b468d926868897c7c800c385dd5025fda8e6782bd17f87125f8254850b"} Oct 29 21:42:25.743969 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:25.743589 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rcs9l" event={"ID":"e69b8cc6-b91f-4cf6-9b3c-92324e5be0e8","Type":"ContainerStarted","Data":"494a9ce75cbd4dcb173495cee2517327d833115b9db840a6a3cbcfe7967521c4"} Oct 29 21:42:25.750454 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:25.750372 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:25.852951 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:25.852868 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:25.853205 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:25.853130 4952 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 21:42:25.927854 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:25.927827 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:26.055780 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:26.055753 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:26.056331 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:26.056315 4952 scope.go:122] "RemoveContainer" containerID="f5da38fd3443febad62ffd6fbe836a07bb27d50ec1618bd0d73ae93c414856ad" Oct 29 21:42:26.056562 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:26.056539 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7jz5d_openshift-console-operator(20070f80-4b65-40ff-9646-e8b7fef85cc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" podUID="20070f80-4b65-40ff-9646-e8b7fef85cc1" Oct 29 21:42:26.232703 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:26.232663 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:26.748614 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:26.748579 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" event={"ID":"53ff731b-5c02-4366-b6b2-c5ecfdc8be38","Type":"ContainerStarted","Data":"0a152624f34fc8219e2c2be64b5185c13c6e460027449af7ce35e6d020b365be"} Oct 29 21:42:26.749122 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:26.748642 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-46z64" event={"ID":"53ff731b-5c02-4366-b6b2-c5ecfdc8be38","Type":"ContainerStarted","Data":"3af89c9d991045cd40f2e8c29ad56df89a69c57a86c1b40e0cc686739b268578"} Oct 29 21:42:26.750413 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:26.750389 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmxmn" event={"ID":"b39d65b2-cca6-4f08-bdec-becd6e000c48","Type":"ContainerStarted","Data":"875e65c40c4d69526b8e56c50733afad0cb65ab3f2fc5870e8622093b905b3b3"} Oct 29 21:42:26.750564 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:26.750435 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmxmn" event={"ID":"b39d65b2-cca6-4f08-bdec-becd6e000c48","Type":"ContainerStarted","Data":"9d879be837e7502dcfb9fa0233bd0a1c33147d6e95cb51bfb7847509d7cdbff2"} Oct 29 21:42:26.750564 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:26.750496 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:26.752423 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:26.752401 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8dsqf" event={"ID":"a4dbf16f-ae81-461b-8564-99fce7202e3c","Type":"ContainerStarted","Data":"02ee0c0ab29f47e8d967a0e76f67509a984fa7bb5e136250311e9a7ad36494f0"} Oct 29 21:42:26.752660 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:26.752645 4952 scope.go:122] "RemoveContainer" containerID="f5da38fd3443febad62ffd6fbe836a07bb27d50ec1618bd0d73ae93c414856ad" Oct 29 21:42:27.043602 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:27.043519 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:27.446146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:27.446112 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:27.661647 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:27.661614 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:27.757395 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:27.757327 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 21:42:27.757853 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:27.757394 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" event={"ID":"20070f80-4b65-40ff-9646-e8b7fef85cc1","Type":"ContainerStarted","Data":"b9a8fddda8b6ffbbf15bc6f85c699a13bf0b4b85c714afde580a7f7f3510165c"} Oct 29 21:42:27.758045 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:27.758030 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:27.762698 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:27.762679 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7jz5d" Oct 29 21:42:27.837793 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:27.837764 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:28.016015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:28.015929 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:28.188526 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:28.188489 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:28.438645 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:28.438607 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:29.396262 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:29.396220 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:29.577439 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:29.577401 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:31.517866 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.517832 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:31.517866 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.517874 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:31.518430 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.517918 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:31.518430 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.517936 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:31.520402 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.520371 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"image-registry-748b45bb85-pfdxr\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:31.520531 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.520443 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a06d7-5255-47dc-8ce7-07d113e03ca1-metrics-certs\") pod \"router-default-56bd78965c-kscnl\" (UID: \"059a06d7-5255-47dc-8ce7-07d113e03ca1\") " pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:31.520668 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.520649 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e889f05-b993-4423-9b9c-4673ad6e70d3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zv2wb\" (UID: \"1e889f05-b993-4423-9b9c-4673ad6e70d3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:31.520748 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.520727 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"image-registry-859d6fc448-k9w4q\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:31.567588 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.567546 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:31.571376 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.571359 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:31.572749 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.572727 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" Oct 29 21:42:31.580472 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.580449 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:31.905341 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.905312 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zv2wb"] Oct 29 21:42:31.908209 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:31.908184 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e889f05_b993_4423_9b9c_4673ad6e70d3.slice/crio-7b80915b5fe2efefd69222b765749ebd73932e85d261949ddda12d39c9ef4417 WatchSource:0}: Error finding container 7b80915b5fe2efefd69222b765749ebd73932e85d261949ddda12d39c9ef4417: Status 404 returned error can't find the container with id 7b80915b5fe2efefd69222b765749ebd73932e85d261949ddda12d39c9ef4417 Oct 29 21:42:31.911870 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:31.911850 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f12b29_bfab_4407_8cae_b618a658c23e.slice/crio-cb29352d94c957eb54bbec2d33785305221c16139b22038e5a6a31905bdde40d WatchSource:0}: Error finding container cb29352d94c957eb54bbec2d33785305221c16139b22038e5a6a31905bdde40d: Status 404 returned error can't find the container with id cb29352d94c957eb54bbec2d33785305221c16139b22038e5a6a31905bdde40d Oct 29 21:42:31.918232 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.918207 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-859d6fc448-k9w4q"] Oct 29 21:42:31.921036 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:31.921015 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059a06d7_5255_47dc_8ce7_07d113e03ca1.slice/crio-35a0148f821d6073253e31fd7f3e1380ba3ac99ad48b3227f84ebe3be6aece1b WatchSource:0}: Error finding container 35a0148f821d6073253e31fd7f3e1380ba3ac99ad48b3227f84ebe3be6aece1b: Status 404 returned error can't find the container with id 35a0148f821d6073253e31fd7f3e1380ba3ac99ad48b3227f84ebe3be6aece1b Oct 29 21:42:31.921771 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.921751 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56bd78965c-kscnl"] Oct 29 21:42:31.925444 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:31.925424 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca158db_9c67_4fb5_bc6b_7c15ceee8772.slice/crio-43382c677dca6170dc2e736925f0046ed854f1745bb682983f7f096fa864f9cc WatchSource:0}: Error finding container 43382c677dca6170dc2e736925f0046ed854f1745bb682983f7f096fa864f9cc: Status 404 returned error can't find the container with id 43382c677dca6170dc2e736925f0046ed854f1745bb682983f7f096fa864f9cc Oct 29 21:42:31.930845 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:31.930822 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-748b45bb85-pfdxr"] Oct 29 21:42:32.192939 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.192818 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:32.375249 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.375217 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:32.775784 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.775754 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" event={"ID":"1e889f05-b993-4423-9b9c-4673ad6e70d3","Type":"ContainerStarted","Data":"7b80915b5fe2efefd69222b765749ebd73932e85d261949ddda12d39c9ef4417"} Oct 29 21:42:32.777313 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.777280 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" event={"ID":"91f12b29-bfab-4407-8cae-b618a658c23e","Type":"ContainerStarted","Data":"c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c"} Oct 29 21:42:32.777453 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.777332 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" event={"ID":"91f12b29-bfab-4407-8cae-b618a658c23e","Type":"ContainerStarted","Data":"cb29352d94c957eb54bbec2d33785305221c16139b22038e5a6a31905bdde40d"} Oct 29 21:42:32.777453 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.777359 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:32.778741 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.778713 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" event={"ID":"5ca158db-9c67-4fb5-bc6b-7c15ceee8772","Type":"ContainerStarted","Data":"80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653"} Oct 29 21:42:32.778842 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.778758 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" event={"ID":"5ca158db-9c67-4fb5-bc6b-7c15ceee8772","Type":"ContainerStarted","Data":"43382c677dca6170dc2e736925f0046ed854f1745bb682983f7f096fa864f9cc"} Oct 29 21:42:32.778842 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.778802 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:32.780177 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.780158 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56bd78965c-kscnl" event={"ID":"059a06d7-5255-47dc-8ce7-07d113e03ca1","Type":"ContainerStarted","Data":"d74480db95b90fd86c8c8b086213562b0a0a66d142508ec0232fa122cf3dab8c"} Oct 29 21:42:32.780247 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:32.780187 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56bd78965c-kscnl" event={"ID":"059a06d7-5255-47dc-8ce7-07d113e03ca1","Type":"ContainerStarted","Data":"35a0148f821d6073253e31fd7f3e1380ba3ac99ad48b3227f84ebe3be6aece1b"} Oct 29 21:42:33.238786 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:33.238751 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-859d6fc448-k9w4q"] Oct 29 21:42:33.492323 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:33.492246 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:33.568661 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:33.568633 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:33.571278 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:33.571257 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:33.695224 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:33.695191 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:33.789856 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:33.789770 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zv2wb" event={"ID":"1e889f05-b993-4423-9b9c-4673ad6e70d3","Type":"ContainerStarted","Data":"c859b268c1577b49e2ed5d2e128747c60d9c011531aa981a0ca913706afe5193"} Oct 29 21:42:33.790396 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:33.790375 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:33.791420 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:33.791400 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56bd78965c-kscnl" Oct 29 21:42:35.528430 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:35.528398 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:35.714172 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:35.714138 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:36.377807 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:36.377772 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:36.569301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:36.569261 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:36.760252 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:36.760175 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mmxmn" Oct 29 21:42:37.596576 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:37.596540 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-krcw7" Oct 29 21:42:38.979221 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:38.979189 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvpq9" Oct 29 21:42:39.319525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:39.319426 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:41.824708 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:41.824673 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:42.009450 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.009411 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:42.092823 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.092749 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ft2d5"] Oct 29 21:42:42.132921 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.132876 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.194596 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.194564 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 21:42:42.194737 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.194662 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 21:42:42.204617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.204585 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c5d8661-d9da-45e0-b9d5-4a83096a3985-sys\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.204617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.204618 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c5d8661-d9da-45e0-b9d5-4a83096a3985-root\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.204825 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.204693 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-wtmp\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.204825 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.204733 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swsfz\" (UniqueName: \"kubernetes.io/projected/9c5d8661-d9da-45e0-b9d5-4a83096a3985-kube-api-access-swsfz\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.204825 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.204799 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-accelerators-collector-config\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.204964 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.204829 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-textfile\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.204964 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.204846 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.204964 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.204899 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-tls\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.205060 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.204974 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c5d8661-d9da-45e0-b9d5-4a83096a3985-metrics-client-ca\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.215059 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.215035 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:42.246900 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.246878 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 21:42:42.247133 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.247112 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 21:42:42.247223 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.247140 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dwjqp\"" Oct 29 21:42:42.306316 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.306275 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c5d8661-d9da-45e0-b9d5-4a83096a3985-sys\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.306492 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.306326 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c5d8661-d9da-45e0-b9d5-4a83096a3985-root\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.306492 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.306358 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-wtmp\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.306492 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.306384 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swsfz\" (UniqueName: \"kubernetes.io/projected/9c5d8661-d9da-45e0-b9d5-4a83096a3985-kube-api-access-swsfz\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.306492 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.306411 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-accelerators-collector-config\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.306492 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.306442 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-textfile\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.306492 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.306468 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.307430 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.306511 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-tls\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.307430 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.306537 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c5d8661-d9da-45e0-b9d5-4a83096a3985-metrics-client-ca\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.307430 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.307173 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c5d8661-d9da-45e0-b9d5-4a83096a3985-metrics-client-ca\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.307430 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.307259 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c5d8661-d9da-45e0-b9d5-4a83096a3985-sys\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.307430 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.307304 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c5d8661-d9da-45e0-b9d5-4a83096a3985-root\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.307692 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.307435 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-wtmp\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.308337 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.308309 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-accelerators-collector-config\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.308574 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.308554 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-textfile\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.313323 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.311294 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.313323 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.311741 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c5d8661-d9da-45e0-b9d5-4a83096a3985-node-exporter-tls\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.333335 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.333307 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swsfz\" (UniqueName: \"kubernetes.io/projected/9c5d8661-d9da-45e0-b9d5-4a83096a3985-kube-api-access-swsfz\") pod \"node-exporter-ft2d5\" (UID: \"9c5d8661-d9da-45e0-b9d5-4a83096a3985\") " pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.390990 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.390873 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:42.481873 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.481842 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ft2d5" Oct 29 21:42:42.517430 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:42:42.517395 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5d8661_d9da_45e0_b9d5_4a83096a3985.slice/crio-1f3436f61507f7a3eba421dd9c816ed87307857d04216f4f63efeb6c839c2880 WatchSource:0}: Error finding container 1f3436f61507f7a3eba421dd9c816ed87307857d04216f4f63efeb6c839c2880: Status 404 returned error can't find the container with id 1f3436f61507f7a3eba421dd9c816ed87307857d04216f4f63efeb6c839c2880 Oct 29 21:42:42.565797 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.565758 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:42.826074 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:42.826042 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ft2d5" event={"ID":"9c5d8661-d9da-45e0-b9d5-4a83096a3985","Type":"ContainerStarted","Data":"1f3436f61507f7a3eba421dd9c816ed87307857d04216f4f63efeb6c839c2880"} Oct 29 21:42:43.196979 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:43.196815 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:43.388945 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:43.388896 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:43.577108 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:43.577078 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:43.832507 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:43.832474 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ft2d5" event={"ID":"9c5d8661-d9da-45e0-b9d5-4a83096a3985","Type":"ContainerStarted","Data":"4ab59b3135c7f60ba44b2b56acbdf3d657fabe0294bf289ff77a38684579c662"} Oct 29 21:42:44.837716 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:44.837678 4952 generic.go:356] "Generic (PLEG): container finished" podID="9c5d8661-d9da-45e0-b9d5-4a83096a3985" containerID="4ab59b3135c7f60ba44b2b56acbdf3d657fabe0294bf289ff77a38684579c662" exitCode=0 Oct 29 21:42:44.838219 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:44.837763 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ft2d5" event={"ID":"9c5d8661-d9da-45e0-b9d5-4a83096a3985","Type":"ContainerDied","Data":"4ab59b3135c7f60ba44b2b56acbdf3d657fabe0294bf289ff77a38684579c662"} Oct 29 21:42:44.961452 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:44.961422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:45.143709 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:45.143624 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:45.460834 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:45.460749 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:45.525397 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:42:45.525363 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:42:45.631223 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:45.631188 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:45.844037 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:45.843999 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ft2d5" event={"ID":"9c5d8661-d9da-45e0-b9d5-4a83096a3985","Type":"ContainerStarted","Data":"62f99c986c12eb4559cc070d023e306d4d8674d2696c4940866d58c55f519723"} Oct 29 21:42:45.844497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:45.844093 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ft2d5" event={"ID":"9c5d8661-d9da-45e0-b9d5-4a83096a3985","Type":"ContainerStarted","Data":"e930c5dd0ae54cd25c12e60175c87ab09eee021ccadde9a1906c89bf7981304a"} Oct 29 21:42:45.844497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:45.844304 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:45.891220 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:45.891173 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ft2d5" podStartSLOduration=2.706524235 podStartE2EDuration="3.891161095s" totalImagesPullingTime="1.104192696s" totalInitContainerRuntime="80.444164ms" isStatefulPod=true podCreationTimestamp="2026-10-29 21:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:42:45.889918294 +0000 UTC m=+31.364506703" watchObservedRunningTime="2026-10-29 21:42:45.891161095 +0000 UTC m=+31.365749513" Oct 29 21:42:46.885689 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:46.885652 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:47.092269 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:47.092229 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:47.184721 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:47.184640 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-289sc" Oct 29 21:42:48.600811 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:48.600772 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:48.784996 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:48.784952 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:49.243882 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:49.243851 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:49.515637 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:49.515554 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:49.720441 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:49.720401 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:50.536707 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:50.536664 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:50.717883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:50.717846 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:52.605069 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:52.605010 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:52.780974 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:52.780929 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:53.794436 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:53.794408 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:42:54.270244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:54.270210 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:54.468688 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:54.468654 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:54.798399 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:54.798372 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:42:55.592667 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:55.592624 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-748b45bb85-pfdxr"] Oct 29 21:42:55.850586 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:55.850508 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:56.039023 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:56.038947 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:59.224000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:59.223955 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:59.410778 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:59.410739 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:42:59.812073 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:42:59.812026 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" podUID="91f12b29-bfab-4407-8cae-b618a658c23e" containerName="registry" containerID="cri-o://c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c" gracePeriod=30 Oct 29 21:43:00.060224 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.060199 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:43:00.157469 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.157437 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-trusted-ca\") pod \"91f12b29-bfab-4407-8cae-b618a658c23e\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " Oct 29 21:43:00.157469 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.157474 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f12b29-bfab-4407-8cae-b618a658c23e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91f12b29-bfab-4407-8cae-b618a658c23e-ca-trust-extracted\") pod \"91f12b29-bfab-4407-8cae-b618a658c23e\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " Oct 29 21:43:00.157708 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.157504 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") pod \"91f12b29-bfab-4407-8cae-b618a658c23e\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " Oct 29 21:43:00.157708 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.157535 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-installation-pull-secrets\") pod \"91f12b29-bfab-4407-8cae-b618a658c23e\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " Oct 29 21:43:00.157708 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.157616 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-bound-sa-token\") pod \"91f12b29-bfab-4407-8cae-b618a658c23e\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " Oct 29 21:43:00.157708 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.157649 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-registry-certificates\") pod \"91f12b29-bfab-4407-8cae-b618a658c23e\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " Oct 29 21:43:00.157938 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.157715 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-kube-api-access-wv7n9\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-kube-api-access-wv7n9\") pod \"91f12b29-bfab-4407-8cae-b618a658c23e\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " Oct 29 21:43:00.157938 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.157752 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-image-registry-private-configuration\") pod \"91f12b29-bfab-4407-8cae-b618a658c23e\" (UID: \"91f12b29-bfab-4407-8cae-b618a658c23e\") " Oct 29 21:43:00.158059 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.157966 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-trusted-ca" pod "91f12b29-bfab-4407-8cae-b618a658c23e" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:00.158319 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.158280 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-registry-certificates" pod "91f12b29-bfab-4407-8cae-b618a658c23e" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:00.160871 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.160758 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-installation-pull-secrets" pod "91f12b29-bfab-4407-8cae-b618a658c23e" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:00.160871 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.160835 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-kube-api-access-wv7n9" pod "91f12b29-bfab-4407-8cae-b618a658c23e" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e"). InnerVolumeSpecName "kube-api-access-wv7n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:00.161112 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.161087 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-image-registry-private-configuration" pod "91f12b29-bfab-4407-8cae-b618a658c23e" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:00.161173 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.161141 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-bound-sa-token" pod "91f12b29-bfab-4407-8cae-b618a658c23e" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:00.161206 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.161188 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls" pod "91f12b29-bfab-4407-8cae-b618a658c23e" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:00.165389 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.165354 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f12b29-bfab-4407-8cae-b618a658c23e-ca-trust-extracted" pod "91f12b29-bfab-4407-8cae-b618a658c23e" (UID: "91f12b29-bfab-4407-8cae-b618a658c23e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:43:00.259174 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.259138 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv7n9\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-kube-api-access-wv7n9\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:00.259174 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.259172 4952 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-image-registry-private-configuration\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:00.259174 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.259183 4952 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-trusted-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:00.259617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.259192 4952 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91f12b29-bfab-4407-8cae-b618a658c23e-ca-trust-extracted\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:00.259617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.259201 4952 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-registry-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:00.259617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.259210 4952 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91f12b29-bfab-4407-8cae-b618a658c23e-installation-pull-secrets\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:00.259617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.259218 4952 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91f12b29-bfab-4407-8cae-b618a658c23e-bound-sa-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:00.259617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.259228 4952 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91f12b29-bfab-4407-8cae-b618a658c23e-registry-certificates\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:00.469258 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.469174 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:00.692391 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.692359 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:00.906454 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.906420 4952 generic.go:356] "Generic (PLEG): container finished" podID="91f12b29-bfab-4407-8cae-b618a658c23e" containerID="c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c" exitCode=0 Oct 29 21:43:00.906626 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.906467 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" event={"ID":"91f12b29-bfab-4407-8cae-b618a658c23e","Type":"ContainerDied","Data":"c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c"} Oct 29 21:43:00.906626 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.906483 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" Oct 29 21:43:00.906626 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.906514 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859d6fc448-k9w4q" event={"ID":"91f12b29-bfab-4407-8cae-b618a658c23e","Type":"ContainerDied","Data":"cb29352d94c957eb54bbec2d33785305221c16139b22038e5a6a31905bdde40d"} Oct 29 21:43:00.906626 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.906546 4952 scope.go:122] "RemoveContainer" containerID="c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c" Oct 29 21:43:00.914771 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.914750 4952 scope.go:122] "RemoveContainer" containerID="c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c" Oct 29 21:43:00.915052 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:43:00.915027 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c\": container with ID starting with c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c not found: ID does not exist" containerID="c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c" Oct 29 21:43:00.915125 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.915060 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c"} err="failed to get container status \"c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c\": rpc error: code = NotFound desc = could not find container \"c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c\": container with ID starting with c24f8585c64e820758df3eb0a5b8319d1e491aff143ea78a6224d9291a999c7c not found: ID does not exist" Oct 29 21:43:00.976176 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.976147 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-859d6fc448-k9w4q"] Oct 29 21:43:00.988597 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:00.988573 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-859d6fc448-k9w4q"] Oct 29 21:43:02.643014 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:02.642980 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f12b29-bfab-4407-8cae-b618a658c23e" path="/var/lib/kubelet/pods/91f12b29-bfab-4407-8cae-b618a658c23e/volumes" Oct 29 21:43:03.286299 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:03.286264 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:03.483827 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:03.483797 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:04.989472 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:04.989435 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:05.208966 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:05.208928 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:05.551853 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:05.551821 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:05.738628 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:05.738590 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:15.524413 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:43:15.524364 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:43:17.602481 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:17.602441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:17.782356 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:17.782320 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:20.616013 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:20.615947 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" podUID="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" containerName="registry" containerID="cri-o://80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653" gracePeriod=30 Oct 29 21:43:20.855473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:20.855448 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:43:20.983427 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:20.983347 4952 generic.go:356] "Generic (PLEG): container finished" podID="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" containerID="80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653" exitCode=0 Oct 29 21:43:20.983427 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:20.983400 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" Oct 29 21:43:20.983591 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:20.983432 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" event={"ID":"5ca158db-9c67-4fb5-bc6b-7c15ceee8772","Type":"ContainerDied","Data":"80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653"} Oct 29 21:43:20.983591 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:20.983487 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-748b45bb85-pfdxr" event={"ID":"5ca158db-9c67-4fb5-bc6b-7c15ceee8772","Type":"ContainerDied","Data":"43382c677dca6170dc2e736925f0046ed854f1745bb682983f7f096fa864f9cc"} Oct 29 21:43:20.983591 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:20.983518 4952 scope.go:122] "RemoveContainer" containerID="80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653" Oct 29 21:43:20.991183 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:20.991162 4952 scope.go:122] "RemoveContainer" containerID="80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653" Oct 29 21:43:20.991425 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:43:20.991404 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653\": container with ID starting with 80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653 not found: ID does not exist" containerID="80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653" Oct 29 21:43:20.991484 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:20.991434 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653"} err="failed to get container status \"80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653\": rpc error: code = NotFound desc = could not find container \"80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653\": container with ID starting with 80f59d353284fb3c76d0a55ee5ba72c6f6af399da13487957ff1bcbb0d871653 not found: ID does not exist" Oct 29 21:43:21.025594 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.025568 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-certificates\") pod \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " Oct 29 21:43:21.025697 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.025601 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") pod \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " Oct 29 21:43:21.025697 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.025632 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-trusted-ca\") pod \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " Oct 29 21:43:21.025697 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.025665 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-image-registry-private-configuration\") pod \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " Oct 29 21:43:21.025811 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.025725 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-bound-sa-token\") pod \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " Oct 29 21:43:21.025811 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.025780 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-kube-api-access-qhssb\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-kube-api-access-qhssb\") pod \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " Oct 29 21:43:21.025939 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.025819 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-ca-trust-extracted\") pod \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " Oct 29 21:43:21.025939 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.025870 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-installation-pull-secrets\") pod \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\" (UID: \"5ca158db-9c67-4fb5-bc6b-7c15ceee8772\") " Oct 29 21:43:21.026072 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.026050 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-certificates" pod "5ca158db-9c67-4fb5-bc6b-7c15ceee8772" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:21.026238 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.026212 4952 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-certificates\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:21.027182 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.027127 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-trusted-ca" pod "5ca158db-9c67-4fb5-bc6b-7c15ceee8772" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:43:21.028253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.028214 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-installation-pull-secrets" pod "5ca158db-9c67-4fb5-bc6b-7c15ceee8772" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:21.028253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.028240 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-kube-api-access-qhssb" pod "5ca158db-9c67-4fb5-bc6b-7c15ceee8772" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772"). InnerVolumeSpecName "kube-api-access-qhssb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:21.028414 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.028310 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls" pod "5ca158db-9c67-4fb5-bc6b-7c15ceee8772" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:21.028521 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.028499 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-image-registry-private-configuration" pod "5ca158db-9c67-4fb5-bc6b-7c15ceee8772" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:43:21.028575 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.028551 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-bound-sa-token" pod "5ca158db-9c67-4fb5-bc6b-7c15ceee8772" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:43:21.032703 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.032649 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-ca-trust-extracted" pod "5ca158db-9c67-4fb5-bc6b-7c15ceee8772" (UID: "5ca158db-9c67-4fb5-bc6b-7c15ceee8772"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:43:21.127267 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.127228 4952 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-bound-sa-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:21.127267 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.127261 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhssb\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-kube-api-access-qhssb\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:21.127267 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.127272 4952 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-ca-trust-extracted\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:21.127496 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.127281 4952 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-installation-pull-secrets\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:21.127496 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.127292 4952 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-registry-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:21.127496 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.127301 4952 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-trusted-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:21.127496 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.127310 4952 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ca158db-9c67-4fb5-bc6b-7c15ceee8772-image-registry-private-configuration\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:43:21.330337 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.330305 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-748b45bb85-pfdxr"] Oct 29 21:43:21.343099 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:21.343064 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-748b45bb85-pfdxr"] Oct 29 21:43:22.640685 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:22.640650 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" path="/var/lib/kubelet/pods/5ca158db-9c67-4fb5-bc6b-7c15ceee8772/volumes" Oct 29 21:43:26.007243 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.007210 4952 generic.go:356] "Generic (PLEG): container finished" podID="1f4b9c11-19cd-46b9-8096-5c9375fa7e91" containerID="6a657c3b0fd89cd5726b0d6c81aa459ceb79fe911b016815216432cf1ec1e827" exitCode=0 Oct 29 21:43:26.007735 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.007292 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mvwvt" event={"ID":"1f4b9c11-19cd-46b9-8096-5c9375fa7e91","Type":"ContainerDied","Data":"6a657c3b0fd89cd5726b0d6c81aa459ceb79fe911b016815216432cf1ec1e827"} Oct 29 21:43:26.007735 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.007620 4952 scope.go:122] "RemoveContainer" containerID="6a657c3b0fd89cd5726b0d6c81aa459ceb79fe911b016815216432cf1ec1e827" Oct 29 21:43:26.721349 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.721317 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-9hgjk-nqfwt"] Oct 29 21:43:26.721605 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.721592 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" containerName="registry" Oct 29 21:43:26.721685 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.721606 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" containerName="registry" Oct 29 21:43:26.721685 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.721620 4952 cpu_manager.go:414] "containerMap: removing container" podUID="91f12b29-bfab-4407-8cae-b618a658c23e" containerName="registry" Oct 29 21:43:26.721685 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.721626 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f12b29-bfab-4407-8cae-b618a658c23e" containerName="registry" Oct 29 21:43:26.721776 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.721697 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca158db-9c67-4fb5-bc6b-7c15ceee8772" containerName="registry" Oct 29 21:43:26.721776 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.721706 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f12b29-bfab-4407-8cae-b618a658c23e" containerName="registry" Oct 29 21:43:26.724599 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.724582 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.744271 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.744247 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9hgjk-nqfwt"] Oct 29 21:43:26.755997 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.755974 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-sjp55\"" Oct 29 21:43:26.763296 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.763276 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-service-ca-bundle\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.763353 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.763311 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd633ba-afcd-4a22-93ea-647ffce3a944-archives-path\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.763353 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.763336 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-trusted-ca-bundle\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.763458 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.763438 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrzd\" (UniqueName: \"kubernetes.io/projected/9bd633ba-afcd-4a22-93ea-647ffce3a944-kube-api-access-9rrzd\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.864612 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.864590 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd633ba-afcd-4a22-93ea-647ffce3a944-archives-path\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.864723 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.864627 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-trusted-ca-bundle\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.864723 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.864656 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrzd\" (UniqueName: \"kubernetes.io/projected/9bd633ba-afcd-4a22-93ea-647ffce3a944-kube-api-access-9rrzd\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.864723 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.864689 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-service-ca-bundle\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.865055 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.865039 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd633ba-afcd-4a22-93ea-647ffce3a944-archives-path\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.865303 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.865288 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-service-ca-bundle\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.865438 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.865421 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-trusted-ca-bundle\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:26.889875 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:26.889851 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrzd\" (UniqueName: \"kubernetes.io/projected/9bd633ba-afcd-4a22-93ea-647ffce3a944-kube-api-access-9rrzd\") pod \"periodic-gathering-9hgjk-nqfwt\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:27.012877 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:27.012798 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mvwvt" event={"ID":"1f4b9c11-19cd-46b9-8096-5c9375fa7e91","Type":"ContainerStarted","Data":"3e4dbda055cf26ddadf2f2fdc7d13bdb2a265e12123c6e695dbc271919101ba6"} Oct 29 21:43:27.035265 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:27.035239 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:43:27.334337 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:27.334310 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9hgjk-nqfwt"] Oct 29 21:43:27.336734 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:43:27.336692 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd633ba_afcd_4a22_93ea_647ffce3a944.slice/crio-8fb923b34d37fe6698694a943893a0d52152a1b372bf933d8c32c1c188850f62 WatchSource:0}: Error finding container 8fb923b34d37fe6698694a943893a0d52152a1b372bf933d8c32c1c188850f62: Status 404 returned error can't find the container with id 8fb923b34d37fe6698694a943893a0d52152a1b372bf933d8c32c1c188850f62 Oct 29 21:43:28.017648 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:28.017612 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" event={"ID":"9bd633ba-afcd-4a22-93ea-647ffce3a944","Type":"ContainerStarted","Data":"0a2e845b6838691f51ed7337fadaae52310489ae52c9636e0543eb64e6294838"} Oct 29 21:43:28.018059 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:28.017672 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" event={"ID":"9bd633ba-afcd-4a22-93ea-647ffce3a944","Type":"ContainerStarted","Data":"8fb923b34d37fe6698694a943893a0d52152a1b372bf933d8c32c1c188850f62"} Oct 29 21:43:28.065579 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:28.065529 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" podStartSLOduration=2.065514751 podStartE2EDuration="2.065514751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:43:28.065005023 +0000 UTC m=+73.539593452" watchObservedRunningTime="2026-10-29 21:43:28.065514751 +0000 UTC m=+73.540103169" Oct 29 21:43:35.544372 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:35.544335 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:35.744068 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:35.744038 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:41.066991 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:41.066956 4952 generic.go:356] "Generic (PLEG): container finished" podID="61ac615e-a093-42ea-92a4-84d5c22c9f41" containerID="4d8df66619ac1dff88fac39bc8625093e69181aa87e5ffc74d97a433c4aeb749" exitCode=0 Oct 29 21:43:41.067354 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:41.067023 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" event={"ID":"61ac615e-a093-42ea-92a4-84d5c22c9f41","Type":"ContainerDied","Data":"4d8df66619ac1dff88fac39bc8625093e69181aa87e5ffc74d97a433c4aeb749"} Oct 29 21:43:41.067354 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:41.067324 4952 scope.go:122] "RemoveContainer" containerID="4d8df66619ac1dff88fac39bc8625093e69181aa87e5ffc74d97a433c4aeb749" Oct 29 21:43:42.072845 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:42.072809 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7gpb5" event={"ID":"61ac615e-a093-42ea-92a4-84d5c22c9f41","Type":"ContainerStarted","Data":"4ac00315abed8f682ab226f26276ca3db832726779c6fd7ff1525c51e8f94d67"} Oct 29 21:43:45.527277 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:43:45.527239 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:43:46.090814 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:46.090775 4952 generic.go:356] "Generic (PLEG): container finished" podID="cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9" containerID="7a09fea05bfd19caa84d02e47d66805bbef77fd0dbe720ba91f00ecb2275b675" exitCode=0 Oct 29 21:43:46.091046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:46.090844 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" event={"ID":"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9","Type":"ContainerDied","Data":"7a09fea05bfd19caa84d02e47d66805bbef77fd0dbe720ba91f00ecb2275b675"} Oct 29 21:43:46.091185 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:46.091170 4952 scope.go:122] "RemoveContainer" containerID="7a09fea05bfd19caa84d02e47d66805bbef77fd0dbe720ba91f00ecb2275b675" Oct 29 21:43:47.104554 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:47.104521 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4zk6" event={"ID":"cdc5b397-30ee-49b3-b52a-00e1fa2b6cf9","Type":"ContainerStarted","Data":"77f101d1f1a84b1c0b865f870cb76917d7d3d22e9baffb4685c78a351d7b3e89"} Oct 29 21:43:52.571071 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:52.571030 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:43:52.762575 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:43:52.762532 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:44:01.238307 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:01.238264 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:44:01.421041 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:01.421008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:44:01.594056 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:01.594023 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:44:05.132890 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:05.132850 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:44:15.524382 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:44:15.524346 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:44:45.524407 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:44:45.524370 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:44:51.333472 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:51.333441 4952 generic.go:356] "Generic (PLEG): container finished" podID="9bd633ba-afcd-4a22-93ea-647ffce3a944" containerID="0a2e845b6838691f51ed7337fadaae52310489ae52c9636e0543eb64e6294838" exitCode=0 Oct 29 21:44:51.333822 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:51.333518 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" event={"ID":"9bd633ba-afcd-4a22-93ea-647ffce3a944","Type":"ContainerDied","Data":"0a2e845b6838691f51ed7337fadaae52310489ae52c9636e0543eb64e6294838"} Oct 29 21:44:52.471183 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.471158 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:44:52.515735 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.515705 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd633ba-afcd-4a22-93ea-647ffce3a944-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd633ba-afcd-4a22-93ea-647ffce3a944-archives-path\") pod \"9bd633ba-afcd-4a22-93ea-647ffce3a944\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " Oct 29 21:44:52.515934 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.515752 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-service-ca-bundle\") pod \"9bd633ba-afcd-4a22-93ea-647ffce3a944\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " Oct 29 21:44:52.515934 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.515783 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-trusted-ca-bundle\") pod \"9bd633ba-afcd-4a22-93ea-647ffce3a944\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " Oct 29 21:44:52.515934 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.515872 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd633ba-afcd-4a22-93ea-647ffce3a944-kube-api-access-9rrzd\" (UniqueName: \"kubernetes.io/projected/9bd633ba-afcd-4a22-93ea-647ffce3a944-kube-api-access-9rrzd\") pod \"9bd633ba-afcd-4a22-93ea-647ffce3a944\" (UID: \"9bd633ba-afcd-4a22-93ea-647ffce3a944\") " Oct 29 21:44:52.516117 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.516074 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd633ba-afcd-4a22-93ea-647ffce3a944-archives-path" pod "9bd633ba-afcd-4a22-93ea-647ffce3a944" (UID: "9bd633ba-afcd-4a22-93ea-647ffce3a944"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:44:52.516246 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.516220 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-service-ca-bundle" pod "9bd633ba-afcd-4a22-93ea-647ffce3a944" (UID: "9bd633ba-afcd-4a22-93ea-647ffce3a944"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:44:52.516321 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.516295 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-trusted-ca-bundle" pod "9bd633ba-afcd-4a22-93ea-647ffce3a944" (UID: "9bd633ba-afcd-4a22-93ea-647ffce3a944"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:44:52.518403 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.518375 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd633ba-afcd-4a22-93ea-647ffce3a944-kube-api-access-9rrzd" pod "9bd633ba-afcd-4a22-93ea-647ffce3a944" (UID: "9bd633ba-afcd-4a22-93ea-647ffce3a944"). InnerVolumeSpecName "kube-api-access-9rrzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:44:52.617021 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.616938 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrzd\" (UniqueName: \"kubernetes.io/projected/9bd633ba-afcd-4a22-93ea-647ffce3a944-kube-api-access-9rrzd\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:44:52.617021 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.616973 4952 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd633ba-afcd-4a22-93ea-647ffce3a944-archives-path\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:44:52.617021 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.616982 4952 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-service-ca-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:44:52.617021 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:52.616991 4952 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd633ba-afcd-4a22-93ea-647ffce3a944-trusted-ca-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:44:53.341858 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:53.341819 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" event={"ID":"9bd633ba-afcd-4a22-93ea-647ffce3a944","Type":"ContainerDied","Data":"8fb923b34d37fe6698694a943893a0d52152a1b372bf933d8c32c1c188850f62"} Oct 29 21:44:53.342044 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:53.341866 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fb923b34d37fe6698694a943893a0d52152a1b372bf933d8c32c1c188850f62" Oct 29 21:44:53.342044 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:53.341832 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9hgjk-nqfwt" Oct 29 21:44:53.854464 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:53.854436 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:44:54.041770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:44:54.041739 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:05.305918 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:05.305863 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:08.446251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:08.446208 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:08.623068 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:08.623040 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:15.397256 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:15.397219 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:15.525520 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:45:15.525485 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:45:15.584832 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:15.584794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:15.758697 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:15.758614 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:21.154672 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:21.154635 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:21.330794 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:21.330754 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:26.855975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:26.855931 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:27.038553 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:27.038515 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:35.538471 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:35.538432 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:35.727479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:45:35.727445 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:45:45.525056 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:45:45.525014 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:46:05.376402 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:46:05.376327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:46:15.524995 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:46:15.524956 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:46:33.017400 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:46:33.017346 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:46:42.482923 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:46:42.482865 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:46:45.525666 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:46:45.525608 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:46:59.588475 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:46:59.588430 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:46:59.783073 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:46:59.783038 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:46:59.966581 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:46:59.966495 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:00.769170 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.769138 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx"] Oct 29 21:47:00.769555 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.769433 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9bd633ba-afcd-4a22-93ea-647ffce3a944" containerName="insights-gathering" Oct 29 21:47:00.769555 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.769443 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd633ba-afcd-4a22-93ea-647ffce3a944" containerName="insights-gathering" Oct 29 21:47:00.769555 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.769495 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd633ba-afcd-4a22-93ea-647ffce3a944" containerName="insights-gathering" Oct 29 21:47:00.772494 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.772477 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:00.799560 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.799524 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 29 21:47:00.799560 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.799545 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 29 21:47:00.799780 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.799759 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-tnh2z\"" Oct 29 21:47:00.800759 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.800740 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx"] Oct 29 21:47:00.801161 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.801146 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blsmt\" (UniqueName: \"kubernetes.io/projected/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-kube-api-access-blsmt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:00.801209 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.801175 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:00.801209 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.801194 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:00.902379 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.902349 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:00.902379 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.902381 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:00.902552 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.902431 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blsmt\" (UniqueName: \"kubernetes.io/projected/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-kube-api-access-blsmt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:00.902731 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.902713 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:00.902776 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.902742 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:00.931018 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:00.930992 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blsmt\" (UniqueName: \"kubernetes.io/projected/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-kube-api-access-blsmt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:01.087692 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:01.087667 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:01.274704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:01.274667 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:01.398974 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:01.398950 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx"] Oct 29 21:47:01.400952 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:01.400925 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3014a71b_1361_48cb_a132_4e1f0c2d6a3c.slice/crio-cf664501552bbcebd42622dbf5da2a55cbee7afd2dcfd38043a64467f3306f84 WatchSource:0}: Error finding container cf664501552bbcebd42622dbf5da2a55cbee7afd2dcfd38043a64467f3306f84: Status 404 returned error can't find the container with id cf664501552bbcebd42622dbf5da2a55cbee7afd2dcfd38043a64467f3306f84 Oct 29 21:47:01.495948 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:01.495892 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:01.796406 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:01.796321 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" event={"ID":"3014a71b-1361-48cb-a132-4e1f0c2d6a3c","Type":"ContainerStarted","Data":"cf664501552bbcebd42622dbf5da2a55cbee7afd2dcfd38043a64467f3306f84"} Oct 29 21:47:02.723891 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:02.723856 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:04.840554 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:04.840519 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6"] Oct 29 21:47:04.845193 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:04.845169 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:04.872690 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:04.872663 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-smfnd\"" Oct 29 21:47:04.874125 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:04.874103 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6"] Oct 29 21:47:04.935272 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:04.935231 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:04.935433 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:04.935364 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:04.935489 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:04.935437 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hcmq\" (UniqueName: \"kubernetes.io/projected/b3181951-f3a5-4d26-824a-e859b4e4ac6b-kube-api-access-2hcmq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:05.036338 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.036304 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:05.036531 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.036371 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hcmq\" (UniqueName: \"kubernetes.io/projected/b3181951-f3a5-4d26-824a-e859b4e4ac6b-kube-api-access-2hcmq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:05.036531 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.036413 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:05.036739 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.036712 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:05.036816 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.036761 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:05.067591 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.067555 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hcmq\" (UniqueName: \"kubernetes.io/projected/b3181951-f3a5-4d26-824a-e859b4e4ac6b-kube-api-access-2hcmq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:05.163267 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.162631 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:05.475297 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.475264 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6"] Oct 29 21:47:05.477836 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:05.477790 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3181951_f3a5_4d26_824a_e859b4e4ac6b.slice/crio-e0b7c5cc998f0e65c8ecae5ba9256f4c925b0cdda7350992976c9f7e48b0d23f WatchSource:0}: Error finding container e0b7c5cc998f0e65c8ecae5ba9256f4c925b0cdda7350992976c9f7e48b0d23f: Status 404 returned error can't find the container with id e0b7c5cc998f0e65c8ecae5ba9256f4c925b0cdda7350992976c9f7e48b0d23f Oct 29 21:47:05.507012 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.506980 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:05.704754 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.704714 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:05.815386 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.815298 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" event={"ID":"b3181951-f3a5-4d26-824a-e859b4e4ac6b","Type":"ContainerStarted","Data":"e0b7c5cc998f0e65c8ecae5ba9256f4c925b0cdda7350992976c9f7e48b0d23f"} Oct 29 21:47:05.913771 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.913734 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd"] Oct 29 21:47:05.918191 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.917586 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:05.934107 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.932884 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd"] Oct 29 21:47:05.943105 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943079 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 21:47:05.943310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943293 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4b7xt\"" Oct 29 21:47:05.943397 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943380 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:05.943438 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943416 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:05.943540 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943504 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:05.943665 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943603 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:05.943665 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943626 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:05.943665 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943652 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:05.943833 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943675 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc99afd-f631-46a3-8d94-bfaae865a10a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:05.943833 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943698 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhlbt\" (UniqueName: \"kubernetes.io/projected/ebc99afd-f631-46a3-8d94-bfaae865a10a-kube-api-access-jhlbt\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:05.943833 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:05.943726 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.038647 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.038613 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz"] Oct 29 21:47:06.042851 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.042337 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.042851 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.042392 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:06.044440 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.044418 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.044567 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.044445 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.044567 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.044461 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.044567 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.044476 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc99afd-f631-46a3-8d94-bfaae865a10a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.044567 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.044531 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhlbt\" (UniqueName: \"kubernetes.io/projected/ebc99afd-f631-46a3-8d94-bfaae865a10a-kube-api-access-jhlbt\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.044782 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.044581 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.044782 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.044640 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.044782 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.044696 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.044782 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.044752 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.045075 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.045004 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.045135 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.045104 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.045191 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.045157 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc99afd-f631-46a3-8d94-bfaae865a10a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.045348 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.045324 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.045490 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.045465 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.047281 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.047257 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.047623 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.047571 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.062450 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.061716 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-w8q9f\"" Oct 29 21:47:06.067537 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.067452 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz"] Oct 29 21:47:06.075167 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.074684 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhlbt\" (UniqueName: \"kubernetes.io/projected/ebc99afd-f631-46a3-8d94-bfaae865a10a-kube-api-access-jhlbt\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.092394 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.092370 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc99afd-f631-46a3-8d94-bfaae865a10a-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-ftwkd\" (UID: \"ebc99afd-f631-46a3-8d94-bfaae865a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.145757 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.145716 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.145935 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.145793 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.145935 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.145828 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b897f2be-4c03-4441-8121-409547c4818d-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.145935 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.145859 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b897f2be-4c03-4441-8121-409547c4818d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.145935 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.145895 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.146152 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.145942 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfn26\" (UniqueName: \"kubernetes.io/projected/b897f2be-4c03-4441-8121-409547c4818d-kube-api-access-rfn26\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.146152 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.146014 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.146234 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.146151 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.146267 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.146217 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b897f2be-4c03-4441-8121-409547c4818d-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.235280 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.235243 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:06.243136 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.243108 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2"] Oct 29 21:47:06.246663 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.246633 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.246779 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.246680 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b897f2be-4c03-4441-8121-409547c4818d-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.246779 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.246710 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.246779 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.246761 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.246970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.246789 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b897f2be-4c03-4441-8121-409547c4818d-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.246970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.246815 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b897f2be-4c03-4441-8121-409547c4818d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.246970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.246851 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.246970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.246883 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfn26\" (UniqueName: \"kubernetes.io/projected/b897f2be-4c03-4441-8121-409547c4818d-kube-api-access-rfn26\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.246970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.246925 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.247220 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.247015 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:06.247485 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.247333 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.247485 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.247406 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.247610 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.247557 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.247707 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.247685 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.248226 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.248200 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b897f2be-4c03-4441-8121-409547c4818d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.250236 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.250211 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b897f2be-4c03-4441-8121-409547c4818d-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.250359 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.250311 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b897f2be-4c03-4441-8121-409547c4818d-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.251657 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.251190 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.268333 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.268307 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-rrfsl\"" Oct 29 21:47:06.269581 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.269559 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2"] Oct 29 21:47:06.277454 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.277427 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b897f2be-4c03-4441-8121-409547c4818d-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.278673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.278652 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfn26\" (UniqueName: \"kubernetes.io/projected/b897f2be-4c03-4441-8121-409547c4818d-kube-api-access-rfn26\") pod \"openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz\" (UID: \"b897f2be-4c03-4441-8121-409547c4818d\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.347947 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.347883 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.348218 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.348025 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.348218 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.348119 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7bc2\" (UniqueName: \"kubernetes.io/projected/da33de69-4e86-485c-8c53-bf455b870bba-kube-api-access-g7bc2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.351425 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.350937 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4"] Oct 29 21:47:06.354426 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.354407 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.370341 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.370322 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-s2v6q\"" Oct 29 21:47:06.378132 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.378106 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4"] Oct 29 21:47:06.401372 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.401344 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:06.450515 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.449250 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.450515 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.449296 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.450515 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.449328 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.450515 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.449376 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7bc2\" (UniqueName: \"kubernetes.io/projected/da33de69-4e86-485c-8c53-bf455b870bba-kube-api-access-g7bc2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.450515 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.449401 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9rrr\" (UniqueName: \"kubernetes.io/projected/330dc71b-1c41-488c-9024-4cc3fd9e46d3-kube-api-access-n9rrr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.450515 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.449428 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.450515 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.449876 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.450515 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.450167 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.457082 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.457049 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts"] Oct 29 21:47:06.468591 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.468290 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.485296 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.485265 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts"] Oct 29 21:47:06.486408 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.486350 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7bc2\" (UniqueName: \"kubernetes.io/projected/da33de69-4e86-485c-8c53-bf455b870bba-kube-api-access-g7bc2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.486628 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.486596 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-bspsm\"" Oct 29 21:47:06.550253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.550216 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.551991 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.551962 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg"] Oct 29 21:47:06.553014 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.552980 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76j9d\" (UniqueName: \"kubernetes.io/projected/11433c8a-25be-4f64-a0d7-44422fd25a73-kube-api-access-76j9d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.553148 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.553100 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9rrr\" (UniqueName: \"kubernetes.io/projected/330dc71b-1c41-488c-9024-4cc3fd9e46d3-kube-api-access-n9rrr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.553241 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.553219 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.553356 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.553263 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.553356 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.553308 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.553463 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.553392 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.553691 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.553665 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.555946 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.555928 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.572620 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.572591 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-cp7kn\"" Oct 29 21:47:06.582371 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.582344 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg"] Oct 29 21:47:06.583075 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.583047 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9rrr\" (UniqueName: \"kubernetes.io/projected/330dc71b-1c41-488c-9024-4cc3fd9e46d3-kube-api-access-n9rrr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.588103 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.586949 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd"] Oct 29 21:47:06.589954 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:06.589927 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc99afd_f631_46a3_8d94_bfaae865a10a.slice/crio-c8732c0da8c96b33c4387485a3010e67d065a852c5dca0b6f6911200baafb2e4 WatchSource:0}: Error finding container c8732c0da8c96b33c4387485a3010e67d065a852c5dca0b6f6911200baafb2e4: Status 404 returned error can't find the container with id c8732c0da8c96b33c4387485a3010e67d065a852c5dca0b6f6911200baafb2e4 Oct 29 21:47:06.593550 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.593392 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:06.654962 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.654548 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76j9d\" (UniqueName: \"kubernetes.io/projected/11433c8a-25be-4f64-a0d7-44422fd25a73-kube-api-access-76j9d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.654962 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.654651 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.654962 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.654672 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.654962 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.654709 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.654962 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.654741 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79z8x\" (UniqueName: \"kubernetes.io/projected/04b33341-64e3-4e16-b946-e192918745fa-kube-api-access-79z8x\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.654962 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.654769 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.656499 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.656474 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.657035 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.657010 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.673063 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.673037 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:06.680811 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.680786 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76j9d\" (UniqueName: \"kubernetes.io/projected/11433c8a-25be-4f64-a0d7-44422fd25a73-kube-api-access-76j9d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.717583 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.717552 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz"] Oct 29 21:47:06.723176 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:06.723139 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb897f2be_4c03_4441_8121_409547c4818d.slice/crio-404bc04a53002ee38aa9b9cb3ab7ebf039e958dabc114a09225ea10214c6e796 WatchSource:0}: Error finding container 404bc04a53002ee38aa9b9cb3ab7ebf039e958dabc114a09225ea10214c6e796: Status 404 returned error can't find the container with id 404bc04a53002ee38aa9b9cb3ab7ebf039e958dabc114a09225ea10214c6e796 Oct 29 21:47:06.755811 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.755784 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.755943 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.755835 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79z8x\" (UniqueName: \"kubernetes.io/projected/04b33341-64e3-4e16-b946-e192918745fa-kube-api-access-79z8x\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.756299 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.756278 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.756744 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.756725 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.756849 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.755866 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.782778 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.782749 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79z8x\" (UniqueName: \"kubernetes.io/projected/04b33341-64e3-4e16-b946-e192918745fa-kube-api-access-79z8x\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.806310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.806284 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:06.822036 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.822002 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" event={"ID":"ebc99afd-f631-46a3-8d94-bfaae865a10a","Type":"ContainerStarted","Data":"c8732c0da8c96b33c4387485a3010e67d065a852c5dca0b6f6911200baafb2e4"} Oct 29 21:47:06.823465 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.823423 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" event={"ID":"b897f2be-4c03-4441-8121-409547c4818d","Type":"ContainerStarted","Data":"404bc04a53002ee38aa9b9cb3ab7ebf039e958dabc114a09225ea10214c6e796"} Oct 29 21:47:06.875032 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.874994 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:06.905977 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.905952 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2"] Oct 29 21:47:06.910843 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:06.910814 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda33de69_4e86_485c_8c53_bf455b870bba.slice/crio-86ae982f172ea90205e0be5b8115ca7331a97d16fc8a59cfbccd2d609f5749c1 WatchSource:0}: Error finding container 86ae982f172ea90205e0be5b8115ca7331a97d16fc8a59cfbccd2d609f5749c1: Status 404 returned error can't find the container with id 86ae982f172ea90205e0be5b8115ca7331a97d16fc8a59cfbccd2d609f5749c1 Oct 29 21:47:06.985990 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:06.984458 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4"] Oct 29 21:47:06.988442 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:06.988413 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330dc71b_1c41_488c_9024_4cc3fd9e46d3.slice/crio-d73e5ec96c5c949dcc794455c3d5c3513ce4ca4bf8d3d28ca1f57c112bb1ce99 WatchSource:0}: Error finding container d73e5ec96c5c949dcc794455c3d5c3513ce4ca4bf8d3d28ca1f57c112bb1ce99: Status 404 returned error can't find the container with id d73e5ec96c5c949dcc794455c3d5c3513ce4ca4bf8d3d28ca1f57c112bb1ce99 Oct 29 21:47:07.117076 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.117005 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts"] Oct 29 21:47:07.120831 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:07.120802 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11433c8a_25be_4f64_a0d7_44422fd25a73.slice/crio-283b0842dd660f3babb993231d3f9c6088030471d268e78d2ceb42f9045bfc46 WatchSource:0}: Error finding container 283b0842dd660f3babb993231d3f9c6088030471d268e78d2ceb42f9045bfc46: Status 404 returned error can't find the container with id 283b0842dd660f3babb993231d3f9c6088030471d268e78d2ceb42f9045bfc46 Oct 29 21:47:07.210768 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.210737 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg"] Oct 29 21:47:07.222648 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:07.214339 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b33341_64e3_4e16_b946_e192918745fa.slice/crio-8808c78222b28e73b857259ada9ceef837195f8bc10fcae341211b0c9301ad82 WatchSource:0}: Error finding container 8808c78222b28e73b857259ada9ceef837195f8bc10fcae341211b0c9301ad82: Status 404 returned error can't find the container with id 8808c78222b28e73b857259ada9ceef837195f8bc10fcae341211b0c9301ad82 Oct 29 21:47:07.838808 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.837772 4952 generic.go:356] "Generic (PLEG): container finished" podID="04b33341-64e3-4e16-b946-e192918745fa" containerID="7ad2ce0aba66e3ad7cb3a37d7c6ef2078f302c1cd95daaa6ea5c288d25425d1c" exitCode=0 Oct 29 21:47:07.838808 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.837871 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" event={"ID":"04b33341-64e3-4e16-b946-e192918745fa","Type":"ContainerDied","Data":"7ad2ce0aba66e3ad7cb3a37d7c6ef2078f302c1cd95daaa6ea5c288d25425d1c"} Oct 29 21:47:07.838808 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.837943 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" event={"ID":"04b33341-64e3-4e16-b946-e192918745fa","Type":"ContainerStarted","Data":"8808c78222b28e73b857259ada9ceef837195f8bc10fcae341211b0c9301ad82"} Oct 29 21:47:07.845781 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.845011 4952 generic.go:356] "Generic (PLEG): container finished" podID="11433c8a-25be-4f64-a0d7-44422fd25a73" containerID="b5f121d6e4fd76ad7e0435336df817ee555a9aba6ecb86e85e7cdc0e51782316" exitCode=0 Oct 29 21:47:07.845781 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.845106 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" event={"ID":"11433c8a-25be-4f64-a0d7-44422fd25a73","Type":"ContainerDied","Data":"b5f121d6e4fd76ad7e0435336df817ee555a9aba6ecb86e85e7cdc0e51782316"} Oct 29 21:47:07.845781 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.845149 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" event={"ID":"11433c8a-25be-4f64-a0d7-44422fd25a73","Type":"ContainerStarted","Data":"283b0842dd660f3babb993231d3f9c6088030471d268e78d2ceb42f9045bfc46"} Oct 29 21:47:07.848850 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.848772 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" event={"ID":"330dc71b-1c41-488c-9024-4cc3fd9e46d3","Type":"ContainerStarted","Data":"d73e5ec96c5c949dcc794455c3d5c3513ce4ca4bf8d3d28ca1f57c112bb1ce99"} Oct 29 21:47:07.853113 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.853033 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" event={"ID":"da33de69-4e86-485c-8c53-bf455b870bba","Type":"ContainerStarted","Data":"86ae982f172ea90205e0be5b8115ca7331a97d16fc8a59cfbccd2d609f5749c1"} Oct 29 21:47:07.858073 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.858048 4952 generic.go:356] "Generic (PLEG): container finished" podID="3014a71b-1361-48cb-a132-4e1f0c2d6a3c" containerID="fb7f6012658b9bd588fa8ea4cde9a0a388b9019681857045ee283cae006f8d98" exitCode=0 Oct 29 21:47:07.858185 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:07.858101 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" event={"ID":"3014a71b-1361-48cb-a132-4e1f0c2d6a3c","Type":"ContainerDied","Data":"fb7f6012658b9bd588fa8ea4cde9a0a388b9019681857045ee283cae006f8d98"} Oct 29 21:47:08.254051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.254018 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf"] Oct 29 21:47:08.257826 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.257795 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.279120 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.279081 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf"] Oct 29 21:47:08.279492 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.279471 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.279593 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.279522 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb2df\" (UniqueName: \"kubernetes.io/projected/c530300a-7545-46d0-b60f-0b3c04fa0b58-kube-api-access-vb2df\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.279593 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.279567 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.280690 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.280643 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-wmwc7\"" Oct 29 21:47:08.380794 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.380710 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2df\" (UniqueName: \"kubernetes.io/projected/c530300a-7545-46d0-b60f-0b3c04fa0b58-kube-api-access-vb2df\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.380794 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.380773 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.381050 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.380876 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.381266 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.381240 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.381487 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.381468 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.417251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.417222 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb2df\" (UniqueName: \"kubernetes.io/projected/c530300a-7545-46d0-b60f-0b3c04fa0b58-kube-api-access-vb2df\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.580740 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.580514 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:08.635497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.634450 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9"] Oct 29 21:47:08.640222 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.639411 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:08.669625 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.669555 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9"] Oct 29 21:47:08.683406 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.683379 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:08.683536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.683512 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5z8\" (UniqueName: \"kubernetes.io/projected/9393281e-e858-4be0-a5b9-af7159487b40-kube-api-access-dw5z8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:08.683636 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.683619 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:08.785413 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.784900 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5z8\" (UniqueName: \"kubernetes.io/projected/9393281e-e858-4be0-a5b9-af7159487b40-kube-api-access-dw5z8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:08.785413 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.785018 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:08.785413 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.785067 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:08.785413 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.785377 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:08.785672 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.785580 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:08.811167 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.811111 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5z8\" (UniqueName: \"kubernetes.io/projected/9393281e-e858-4be0-a5b9-af7159487b40-kube-api-access-dw5z8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:08.869469 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.869437 4952 generic.go:356] "Generic (PLEG): container finished" podID="04b33341-64e3-4e16-b946-e192918745fa" containerID="45240bb593e9ea23b33f37a0165c6d4b2665880a5739b3e8f59c0ac5d563f8c2" exitCode=0 Oct 29 21:47:08.869606 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.869526 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" event={"ID":"04b33341-64e3-4e16-b946-e192918745fa","Type":"ContainerDied","Data":"45240bb593e9ea23b33f37a0165c6d4b2665880a5739b3e8f59c0ac5d563f8c2"} Oct 29 21:47:08.873469 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.873440 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" event={"ID":"11433c8a-25be-4f64-a0d7-44422fd25a73","Type":"ContainerStarted","Data":"a3d546b84a6af8f00056c958a9d5db4711040d32f42400eb90cb9470178e6d7e"} Oct 29 21:47:08.877734 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.877707 4952 generic.go:356] "Generic (PLEG): container finished" podID="3014a71b-1361-48cb-a132-4e1f0c2d6a3c" containerID="12b711024bdd73e9d681f6bb302b400c5c352c90cf482b127a2a5def8f3632a3" exitCode=0 Oct 29 21:47:08.877824 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.877761 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" event={"ID":"3014a71b-1361-48cb-a132-4e1f0c2d6a3c","Type":"ContainerDied","Data":"12b711024bdd73e9d681f6bb302b400c5c352c90cf482b127a2a5def8f3632a3"} Oct 29 21:47:08.917382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.917348 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf"] Oct 29 21:47:08.939214 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:08.939172 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc530300a_7545_46d0_b60f_0b3c04fa0b58.slice/crio-699e2c1db13de2bed2185aaff047a757933048e48f39fe1659720359ec25e5de WatchSource:0}: Error finding container 699e2c1db13de2bed2185aaff047a757933048e48f39fe1659720359ec25e5de: Status 404 returned error can't find the container with id 699e2c1db13de2bed2185aaff047a757933048e48f39fe1659720359ec25e5de Oct 29 21:47:08.995583 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:08.995547 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:09.324316 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:09.324266 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9"] Oct 29 21:47:09.330855 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:09.330826 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9393281e_e858_4be0_a5b9_af7159487b40.slice/crio-be3578b1e6ba989c05a918343a0adb0614581883f9dbfbfbe144b170f7bac413 WatchSource:0}: Error finding container be3578b1e6ba989c05a918343a0adb0614581883f9dbfbfbe144b170f7bac413: Status 404 returned error can't find the container with id be3578b1e6ba989c05a918343a0adb0614581883f9dbfbfbe144b170f7bac413 Oct 29 21:47:09.808123 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:09.808082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:09.892845 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:09.892811 4952 generic.go:356] "Generic (PLEG): container finished" podID="c530300a-7545-46d0-b60f-0b3c04fa0b58" containerID="78e72c55b065b02a043b98cc46a00cdc0379a289a2492ff388743d623f2aa514" exitCode=0 Oct 29 21:47:09.893046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:09.892937 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" event={"ID":"c530300a-7545-46d0-b60f-0b3c04fa0b58","Type":"ContainerDied","Data":"78e72c55b065b02a043b98cc46a00cdc0379a289a2492ff388743d623f2aa514"} Oct 29 21:47:09.893046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:09.892991 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" event={"ID":"c530300a-7545-46d0-b60f-0b3c04fa0b58","Type":"ContainerStarted","Data":"699e2c1db13de2bed2185aaff047a757933048e48f39fe1659720359ec25e5de"} Oct 29 21:47:09.896981 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:09.896957 4952 generic.go:356] "Generic (PLEG): container finished" podID="11433c8a-25be-4f64-a0d7-44422fd25a73" containerID="a3d546b84a6af8f00056c958a9d5db4711040d32f42400eb90cb9470178e6d7e" exitCode=0 Oct 29 21:47:09.897347 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:09.897321 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" event={"ID":"11433c8a-25be-4f64-a0d7-44422fd25a73","Type":"ContainerDied","Data":"a3d546b84a6af8f00056c958a9d5db4711040d32f42400eb90cb9470178e6d7e"} Oct 29 21:47:09.905578 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:09.905554 4952 generic.go:356] "Generic (PLEG): container finished" podID="9393281e-e858-4be0-a5b9-af7159487b40" containerID="17dbc9d957df0bd09bd9c4263593a354a14729d8e4ce8f8281265bf034bcb7cf" exitCode=0 Oct 29 21:47:09.905674 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:09.905614 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" event={"ID":"9393281e-e858-4be0-a5b9-af7159487b40","Type":"ContainerDied","Data":"17dbc9d957df0bd09bd9c4263593a354a14729d8e4ce8f8281265bf034bcb7cf"} Oct 29 21:47:09.905674 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:09.905648 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" event={"ID":"9393281e-e858-4be0-a5b9-af7159487b40","Type":"ContainerStarted","Data":"be3578b1e6ba989c05a918343a0adb0614581883f9dbfbfbe144b170f7bac413"} Oct 29 21:47:10.932745 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:10.932663 4952 generic.go:356] "Generic (PLEG): container finished" podID="9393281e-e858-4be0-a5b9-af7159487b40" containerID="eed0577ea70778c060692ccac48eddc15a0d716f237072568750ef6dadac0c57" exitCode=0 Oct 29 21:47:10.932745 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:10.932743 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" event={"ID":"9393281e-e858-4be0-a5b9-af7159487b40","Type":"ContainerDied","Data":"eed0577ea70778c060692ccac48eddc15a0d716f237072568750ef6dadac0c57"} Oct 29 21:47:12.750621 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:12.750060 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:47:12.750621 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:12.750189 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:47:12.750621 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:12.750249 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:47:12.927340 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:12.927304 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:12.971995 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:12.971956 4952 generic.go:356] "Generic (PLEG): container finished" podID="b3181951-f3a5-4d26-824a-e859b4e4ac6b" containerID="23e9d5fef64ef20c002d98f0ece92615bbdc4d5ae46d6fbb04f226f0ec885373" exitCode=0 Oct 29 21:47:12.972173 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:12.972042 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" event={"ID":"b3181951-f3a5-4d26-824a-e859b4e4ac6b","Type":"ContainerDied","Data":"23e9d5fef64ef20c002d98f0ece92615bbdc4d5ae46d6fbb04f226f0ec885373"} Oct 29 21:47:12.977759 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:12.977725 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" event={"ID":"ebc99afd-f631-46a3-8d94-bfaae865a10a","Type":"ContainerStarted","Data":"1f4fe74fd10c390eeaa683387f563eea6b1ff82441ba5b09f77326e67058b923"} Oct 29 21:47:13.067780 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:13.067042 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" podStartSLOduration=1.910480546 podStartE2EDuration="8.067022772s" totalImagesPullingTime="6.156542226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:13.065602386 +0000 UTC m=+298.540190805" watchObservedRunningTime="2026-10-29 21:47:13.067022772 +0000 UTC m=+298.541611186" Oct 29 21:47:13.237222 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:13.237180 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:13.239090 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:13.239057 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ftwkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Oct 29 21:47:13.239216 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:13.239123 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" podUID="ebc99afd-f631-46a3-8d94-bfaae865a10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Oct 29 21:47:14.242285 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:14.242039 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:14.742982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:14.742951 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 21:47:14.744659 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:14.744638 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 21:47:14.994066 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:14.993807 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:14.994648 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:14.994625 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:47:15.540578 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:47:15.540434 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:47:16.002412 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:16.001542 4952 generic.go:356] "Generic (PLEG): container finished" podID="330dc71b-1c41-488c-9024-4cc3fd9e46d3" containerID="9e7f5ca8f95a85fd75bc2da18845e5bd648e838821601f8dbf298918a5725300" exitCode=0 Oct 29 21:47:16.002412 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:16.001650 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" event={"ID":"330dc71b-1c41-488c-9024-4cc3fd9e46d3","Type":"ContainerDied","Data":"9e7f5ca8f95a85fd75bc2da18845e5bd648e838821601f8dbf298918a5725300"} Oct 29 21:47:16.005980 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:16.005259 4952 generic.go:356] "Generic (PLEG): container finished" podID="da33de69-4e86-485c-8c53-bf455b870bba" containerID="11d15dc70ed651fc57090d039bc7b0d634832534e0514b6af3df7defc003b279" exitCode=0 Oct 29 21:47:16.005980 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:16.005422 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" event={"ID":"da33de69-4e86-485c-8c53-bf455b870bba","Type":"ContainerDied","Data":"11d15dc70ed651fc57090d039bc7b0d634832534e0514b6af3df7defc003b279"} Oct 29 21:47:16.009776 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:16.009747 4952 generic.go:356] "Generic (PLEG): container finished" podID="c530300a-7545-46d0-b60f-0b3c04fa0b58" containerID="322e4eaa347c6a026ffc79d23aa6a5eef0090c10d033d343d3518b5bc427b41e" exitCode=0 Oct 29 21:47:16.010832 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:16.010809 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" event={"ID":"c530300a-7545-46d0-b60f-0b3c04fa0b58","Type":"ContainerDied","Data":"322e4eaa347c6a026ffc79d23aa6a5eef0090c10d033d343d3518b5bc427b41e"} Oct 29 21:47:17.021330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:17.021248 4952 generic.go:356] "Generic (PLEG): container finished" podID="330dc71b-1c41-488c-9024-4cc3fd9e46d3" containerID="b8039f09131e3f80cc6f06f900ef943b9674b9042348aacb4b61b2ab8723f3db" exitCode=0 Oct 29 21:47:17.021775 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:17.021339 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" event={"ID":"330dc71b-1c41-488c-9024-4cc3fd9e46d3","Type":"ContainerDied","Data":"b8039f09131e3f80cc6f06f900ef943b9674b9042348aacb4b61b2ab8723f3db"} Oct 29 21:47:17.028838 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:17.027528 4952 generic.go:356] "Generic (PLEG): container finished" podID="da33de69-4e86-485c-8c53-bf455b870bba" containerID="8e8a1b226738bad0a75f770ac0cbf4e4c4dc09c25729caade24a970b98f8cfaa" exitCode=0 Oct 29 21:47:17.028838 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:17.028778 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" event={"ID":"da33de69-4e86-485c-8c53-bf455b870bba","Type":"ContainerDied","Data":"8e8a1b226738bad0a75f770ac0cbf4e4c4dc09c25729caade24a970b98f8cfaa"} Oct 29 21:47:18.051823 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:18.051710 4952 generic.go:356] "Generic (PLEG): container finished" podID="b3181951-f3a5-4d26-824a-e859b4e4ac6b" containerID="34859756f6932338e275b658a3178889c8ac168fcd4c377fccab5e58aab2b61a" exitCode=0 Oct 29 21:47:18.052308 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:18.051838 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" event={"ID":"b3181951-f3a5-4d26-824a-e859b4e4ac6b","Type":"ContainerDied","Data":"34859756f6932338e275b658a3178889c8ac168fcd4c377fccab5e58aab2b61a"} Oct 29 21:47:18.900758 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:18.900719 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:47:18.900880 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:18.900812 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:47:18.900880 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:18.900851 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:47:19.070339 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:19.069448 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" event={"ID":"b897f2be-4c03-4441-8121-409547c4818d","Type":"ContainerStarted","Data":"1af5366722494418811a7154a71a3e08f451bd63a12495f0aa32d656e1b36e85"} Oct 29 21:47:19.402876 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:19.402377 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:20.408866 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:20.408829 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:20.482658 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:20.482597 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" podStartSLOduration=2.307795046 podStartE2EDuration="14.482582019s" totalImagesPullingTime="12.174786973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:19.134088542 +0000 UTC m=+304.608676961" watchObservedRunningTime="2026-10-29 21:47:20.482582019 +0000 UTC m=+305.957170437" Oct 29 21:47:21.091745 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:21.091712 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:21.093210 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:21.092995 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz" Oct 29 21:47:24.989034 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:24.988976 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:25.175034 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:25.174940 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:26.141969 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:26.137968 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" event={"ID":"04b33341-64e3-4e16-b946-e192918745fa","Type":"ContainerStarted","Data":"1bec2de666db2a35db3138271e140f01f43f8ea47a1aaf4dbbad23a8c5c5f588"} Oct 29 21:47:26.144553 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:26.144526 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" event={"ID":"c530300a-7545-46d0-b60f-0b3c04fa0b58","Type":"ContainerStarted","Data":"5f23dd052982e82abcd425260f67a4122839be40073b20d80f89e1ff0454cf4d"} Oct 29 21:47:26.191217 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:26.191166 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" podStartSLOduration=2.348751285 podStartE2EDuration="20.191151728s" totalImagesPullingTime="17.842400443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:26.189421896 +0000 UTC m=+311.664010315" watchObservedRunningTime="2026-10-29 21:47:26.191151728 +0000 UTC m=+311.665740150" Oct 29 21:47:26.238969 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:26.238891 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" podStartSLOduration=3.08565571 podStartE2EDuration="18.238869729s" totalImagesPullingTime="15.153214019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:26.236507048 +0000 UTC m=+311.711095467" watchObservedRunningTime="2026-10-29 21:47:26.238869729 +0000 UTC m=+311.713458149" Oct 29 21:47:27.152102 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:27.152069 4952 generic.go:356] "Generic (PLEG): container finished" podID="04b33341-64e3-4e16-b946-e192918745fa" containerID="1bec2de666db2a35db3138271e140f01f43f8ea47a1aaf4dbbad23a8c5c5f588" exitCode=0 Oct 29 21:47:27.152437 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:27.152150 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" event={"ID":"04b33341-64e3-4e16-b946-e192918745fa","Type":"ContainerDied","Data":"1bec2de666db2a35db3138271e140f01f43f8ea47a1aaf4dbbad23a8c5c5f588"} Oct 29 21:47:27.154143 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:27.154121 4952 generic.go:356] "Generic (PLEG): container finished" podID="c530300a-7545-46d0-b60f-0b3c04fa0b58" containerID="5f23dd052982e82abcd425260f67a4122839be40073b20d80f89e1ff0454cf4d" exitCode=0 Oct 29 21:47:27.154263 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:27.154202 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" event={"ID":"c530300a-7545-46d0-b60f-0b3c04fa0b58","Type":"ContainerDied","Data":"5f23dd052982e82abcd425260f67a4122839be40073b20d80f89e1ff0454cf4d"} Oct 29 21:47:27.156246 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:27.156225 4952 generic.go:356] "Generic (PLEG): container finished" podID="11433c8a-25be-4f64-a0d7-44422fd25a73" containerID="f523e6713b79cb108c24d419d1ebf9317e8645c186819bfdcec37223fd951601" exitCode=0 Oct 29 21:47:27.156320 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:27.156303 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" event={"ID":"11433c8a-25be-4f64-a0d7-44422fd25a73","Type":"ContainerDied","Data":"f523e6713b79cb108c24d419d1ebf9317e8645c186819bfdcec37223fd951601"} Oct 29 21:47:27.158323 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:27.158302 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" event={"ID":"da33de69-4e86-485c-8c53-bf455b870bba","Type":"ContainerStarted","Data":"c457dbb0b359c124f25c3d2f0d10ba404556d86ff5ecc9aba33d4931bda31226"} Oct 29 21:47:27.160386 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:27.160365 4952 generic.go:356] "Generic (PLEG): container finished" podID="3014a71b-1361-48cb-a132-4e1f0c2d6a3c" containerID="8bbe1bd6c419a9d47a713bc6bc7facc8282fea579742300b94671348988af0a1" exitCode=0 Oct 29 21:47:27.160493 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:27.160416 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" event={"ID":"3014a71b-1361-48cb-a132-4e1f0c2d6a3c","Type":"ContainerDied","Data":"8bbe1bd6c419a9d47a713bc6bc7facc8282fea579742300b94671348988af0a1"} Oct 29 21:47:27.370935 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:27.370825 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" podStartSLOduration=2.693239337 podStartE2EDuration="21.370808383s" totalImagesPullingTime="18.677569046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:27.370046938 +0000 UTC m=+312.844635356" watchObservedRunningTime="2026-10-29 21:47:27.370808383 +0000 UTC m=+312.845396795" Oct 29 21:47:28.168294 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.168196 4952 generic.go:356] "Generic (PLEG): container finished" podID="330dc71b-1c41-488c-9024-4cc3fd9e46d3" containerID="24eeaab739df15bf591918039d7ab96a571f87724925c3775d088026b94bcb03" exitCode=0 Oct 29 21:47:28.168688 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.168287 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" event={"ID":"330dc71b-1c41-488c-9024-4cc3fd9e46d3","Type":"ContainerDied","Data":"24eeaab739df15bf591918039d7ab96a571f87724925c3775d088026b94bcb03"} Oct 29 21:47:28.170350 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.170324 4952 generic.go:356] "Generic (PLEG): container finished" podID="da33de69-4e86-485c-8c53-bf455b870bba" containerID="c457dbb0b359c124f25c3d2f0d10ba404556d86ff5ecc9aba33d4931bda31226" exitCode=0 Oct 29 21:47:28.170435 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.170415 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" event={"ID":"da33de69-4e86-485c-8c53-bf455b870bba","Type":"ContainerDied","Data":"c457dbb0b359c124f25c3d2f0d10ba404556d86ff5ecc9aba33d4931bda31226"} Oct 29 21:47:28.172302 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.172278 4952 generic.go:356] "Generic (PLEG): container finished" podID="9393281e-e858-4be0-a5b9-af7159487b40" containerID="78624316635eb953d0dfb24c8bf55172afb72cee596eed16d90a7db8d3dd3c14" exitCode=0 Oct 29 21:47:28.172394 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.172356 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" event={"ID":"9393281e-e858-4be0-a5b9-af7159487b40","Type":"ContainerDied","Data":"78624316635eb953d0dfb24c8bf55172afb72cee596eed16d90a7db8d3dd3c14"} Oct 29 21:47:28.174423 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.174403 4952 generic.go:356] "Generic (PLEG): container finished" podID="b3181951-f3a5-4d26-824a-e859b4e4ac6b" containerID="42ddfd4242b48a11def282227bc37ba1bf1e5a5ae35e01af3a57416b04a12f32" exitCode=0 Oct 29 21:47:28.174527 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.174502 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" event={"ID":"b3181951-f3a5-4d26-824a-e859b4e4ac6b","Type":"ContainerDied","Data":"42ddfd4242b48a11def282227bc37ba1bf1e5a5ae35e01af3a57416b04a12f32"} Oct 29 21:47:28.353834 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.353808 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:28.379984 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.379958 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:28.396239 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.396217 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:28.410117 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.410098 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:28.492074 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.491996 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-bundle\" (UniqueName: \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-bundle\") pod \"11433c8a-25be-4f64-a0d7-44422fd25a73\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " Oct 29 21:47:28.492074 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492029 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-util\" (UniqueName: \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-util\") pod \"c530300a-7545-46d0-b60f-0b3c04fa0b58\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " Oct 29 21:47:28.492074 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492071 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-bundle\") pod \"04b33341-64e3-4e16-b946-e192918745fa\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " Oct 29 21:47:28.492340 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492097 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-util\" (UniqueName: \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-util\") pod \"11433c8a-25be-4f64-a0d7-44422fd25a73\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " Oct 29 21:47:28.492340 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492120 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-kube-api-access-blsmt\" (UniqueName: \"kubernetes.io/projected/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-kube-api-access-blsmt\") pod \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " Oct 29 21:47:28.492340 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492148 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-util\" (UniqueName: \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-util\") pod \"04b33341-64e3-4e16-b946-e192918745fa\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " Oct 29 21:47:28.492340 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492172 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11433c8a-25be-4f64-a0d7-44422fd25a73-kube-api-access-76j9d\" (UniqueName: \"kubernetes.io/projected/11433c8a-25be-4f64-a0d7-44422fd25a73-kube-api-access-76j9d\") pod \"11433c8a-25be-4f64-a0d7-44422fd25a73\" (UID: \"11433c8a-25be-4f64-a0d7-44422fd25a73\") " Oct 29 21:47:28.492340 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492198 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-bundle\") pod \"c530300a-7545-46d0-b60f-0b3c04fa0b58\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " Oct 29 21:47:28.492340 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492258 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-bundle\") pod \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " Oct 29 21:47:28.492340 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492307 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-util\" (UniqueName: \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-util\") pod \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\" (UID: \"3014a71b-1361-48cb-a132-4e1f0c2d6a3c\") " Oct 29 21:47:28.492693 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492348 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b33341-64e3-4e16-b946-e192918745fa-kube-api-access-79z8x\" (UniqueName: \"kubernetes.io/projected/04b33341-64e3-4e16-b946-e192918745fa-kube-api-access-79z8x\") pod \"04b33341-64e3-4e16-b946-e192918745fa\" (UID: \"04b33341-64e3-4e16-b946-e192918745fa\") " Oct 29 21:47:28.492693 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492377 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c530300a-7545-46d0-b60f-0b3c04fa0b58-kube-api-access-vb2df\" (UniqueName: \"kubernetes.io/projected/c530300a-7545-46d0-b60f-0b3c04fa0b58-kube-api-access-vb2df\") pod \"c530300a-7545-46d0-b60f-0b3c04fa0b58\" (UID: \"c530300a-7545-46d0-b60f-0b3c04fa0b58\") " Oct 29 21:47:28.492693 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492520 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-bundle" pod "11433c8a-25be-4f64-a0d7-44422fd25a73" (UID: "11433c8a-25be-4f64-a0d7-44422fd25a73"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:28.492693 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.492662 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.493592 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.493563 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-bundle" pod "3014a71b-1361-48cb-a132-4e1f0c2d6a3c" (UID: "3014a71b-1361-48cb-a132-4e1f0c2d6a3c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:28.493713 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.493608 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-bundle" pod "c530300a-7545-46d0-b60f-0b3c04fa0b58" (UID: "c530300a-7545-46d0-b60f-0b3c04fa0b58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:28.496236 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.496204 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-bundle" pod "04b33341-64e3-4e16-b946-e192918745fa" (UID: "04b33341-64e3-4e16-b946-e192918745fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:28.496880 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.496857 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-kube-api-access-blsmt" pod "3014a71b-1361-48cb-a132-4e1f0c2d6a3c" (UID: "3014a71b-1361-48cb-a132-4e1f0c2d6a3c"). InnerVolumeSpecName "kube-api-access-blsmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:47:28.497046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.497023 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c530300a-7545-46d0-b60f-0b3c04fa0b58-kube-api-access-vb2df" pod "c530300a-7545-46d0-b60f-0b3c04fa0b58" (UID: "c530300a-7545-46d0-b60f-0b3c04fa0b58"). InnerVolumeSpecName "kube-api-access-vb2df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:47:28.497121 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.497099 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b33341-64e3-4e16-b946-e192918745fa-kube-api-access-79z8x" pod "04b33341-64e3-4e16-b946-e192918745fa" (UID: "04b33341-64e3-4e16-b946-e192918745fa"). InnerVolumeSpecName "kube-api-access-79z8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:47:28.498134 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.498102 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11433c8a-25be-4f64-a0d7-44422fd25a73-kube-api-access-76j9d" pod "11433c8a-25be-4f64-a0d7-44422fd25a73" (UID: "11433c8a-25be-4f64-a0d7-44422fd25a73"). InnerVolumeSpecName "kube-api-access-76j9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:47:28.499746 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.499725 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-util" pod "c530300a-7545-46d0-b60f-0b3c04fa0b58" (UID: "c530300a-7545-46d0-b60f-0b3c04fa0b58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:28.499864 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.499845 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-util" pod "3014a71b-1361-48cb-a132-4e1f0c2d6a3c" (UID: "3014a71b-1361-48cb-a132-4e1f0c2d6a3c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:28.500407 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.500382 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-util" pod "11433c8a-25be-4f64-a0d7-44422fd25a73" (UID: "11433c8a-25be-4f64-a0d7-44422fd25a73"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:28.500826 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.500807 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-util" pod "04b33341-64e3-4e16-b946-e192918745fa" (UID: "04b33341-64e3-4e16-b946-e192918745fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:28.594084 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594055 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-util\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.594084 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594081 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79z8x\" (UniqueName: \"kubernetes.io/projected/04b33341-64e3-4e16-b946-e192918745fa-kube-api-access-79z8x\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.594251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594092 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb2df\" (UniqueName: \"kubernetes.io/projected/c530300a-7545-46d0-b60f-0b3c04fa0b58-kube-api-access-vb2df\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.594251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594103 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-util\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.594251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594113 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.594251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594120 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11433c8a-25be-4f64-a0d7-44422fd25a73-util\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.594251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594127 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blsmt\" (UniqueName: \"kubernetes.io/projected/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-kube-api-access-blsmt\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.594251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594135 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b33341-64e3-4e16-b946-e192918745fa-util\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.594251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594143 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76j9d\" (UniqueName: \"kubernetes.io/projected/11433c8a-25be-4f64-a0d7-44422fd25a73-kube-api-access-76j9d\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.594251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594151 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c530300a-7545-46d0-b60f-0b3c04fa0b58-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:28.594251 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:28.594159 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3014a71b-1361-48cb-a132-4e1f0c2d6a3c-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.181120 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.181043 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" Oct 29 21:47:29.181120 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.181071 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xc2hx" event={"ID":"3014a71b-1361-48cb-a132-4e1f0c2d6a3c","Type":"ContainerDied","Data":"cf664501552bbcebd42622dbf5da2a55cbee7afd2dcfd38043a64467f3306f84"} Oct 29 21:47:29.181120 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.181121 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf664501552bbcebd42622dbf5da2a55cbee7afd2dcfd38043a64467f3306f84" Oct 29 21:47:29.183150 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.183126 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" event={"ID":"04b33341-64e3-4e16-b946-e192918745fa","Type":"ContainerDied","Data":"8808c78222b28e73b857259ada9ceef837195f8bc10fcae341211b0c9301ad82"} Oct 29 21:47:29.183311 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.183158 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8sfwg" Oct 29 21:47:29.183311 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.183168 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8808c78222b28e73b857259ada9ceef837195f8bc10fcae341211b0c9301ad82" Oct 29 21:47:29.185464 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.185441 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" event={"ID":"c530300a-7545-46d0-b60f-0b3c04fa0b58","Type":"ContainerDied","Data":"699e2c1db13de2bed2185aaff047a757933048e48f39fe1659720359ec25e5de"} Oct 29 21:47:29.185569 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.185475 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="699e2c1db13de2bed2185aaff047a757933048e48f39fe1659720359ec25e5de" Oct 29 21:47:29.185625 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.185575 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv5kdf" Oct 29 21:47:29.187367 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.187348 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" Oct 29 21:47:29.187471 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.187399 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mxbts" event={"ID":"11433c8a-25be-4f64-a0d7-44422fd25a73","Type":"ContainerDied","Data":"283b0842dd660f3babb993231d3f9c6088030471d268e78d2ceb42f9045bfc46"} Oct 29 21:47:29.187471 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.187437 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="283b0842dd660f3babb993231d3f9c6088030471d268e78d2ceb42f9045bfc46" Oct 29 21:47:29.350253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.350224 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:29.381081 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.381048 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:29.388643 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.388626 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:29.391384 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.391366 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:29.500330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500249 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3181951-f3a5-4d26-824a-e859b4e4ac6b-kube-api-access-2hcmq\" (UniqueName: \"kubernetes.io/projected/b3181951-f3a5-4d26-824a-e859b4e4ac6b-kube-api-access-2hcmq\") pod \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " Oct 29 21:47:29.500330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500310 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-util\" (UniqueName: \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-util\") pod \"da33de69-4e86-485c-8c53-bf455b870bba\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " Oct 29 21:47:29.500540 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500352 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330dc71b-1c41-488c-9024-4cc3fd9e46d3-kube-api-access-n9rrr\" (UniqueName: \"kubernetes.io/projected/330dc71b-1c41-488c-9024-4cc3fd9e46d3-kube-api-access-n9rrr\") pod \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " Oct 29 21:47:29.500540 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500390 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da33de69-4e86-485c-8c53-bf455b870bba-kube-api-access-g7bc2\" (UniqueName: \"kubernetes.io/projected/da33de69-4e86-485c-8c53-bf455b870bba-kube-api-access-g7bc2\") pod \"da33de69-4e86-485c-8c53-bf455b870bba\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " Oct 29 21:47:29.500540 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500421 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9393281e-e858-4be0-a5b9-af7159487b40-kube-api-access-dw5z8\" (UniqueName: \"kubernetes.io/projected/9393281e-e858-4be0-a5b9-af7159487b40-kube-api-access-dw5z8\") pod \"9393281e-e858-4be0-a5b9-af7159487b40\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " Oct 29 21:47:29.500540 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500460 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-bundle\") pod \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " Oct 29 21:47:29.500540 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500482 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-util\" (UniqueName: \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-util\") pod \"9393281e-e858-4be0-a5b9-af7159487b40\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " Oct 29 21:47:29.500540 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500536 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-util\" (UniqueName: \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-util\") pod \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " Oct 29 21:47:29.500815 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500613 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-bundle\") pod \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\" (UID: \"b3181951-f3a5-4d26-824a-e859b4e4ac6b\") " Oct 29 21:47:29.500815 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500655 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-bundle\" (UniqueName: \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-bundle\") pod \"da33de69-4e86-485c-8c53-bf455b870bba\" (UID: \"da33de69-4e86-485c-8c53-bf455b870bba\") " Oct 29 21:47:29.500815 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500696 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-bundle\") pod \"9393281e-e858-4be0-a5b9-af7159487b40\" (UID: \"9393281e-e858-4be0-a5b9-af7159487b40\") " Oct 29 21:47:29.500815 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.500730 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-util\" (UniqueName: \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-util\") pod \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\" (UID: \"330dc71b-1c41-488c-9024-4cc3fd9e46d3\") " Oct 29 21:47:29.502138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.502114 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-bundle" pod "b3181951-f3a5-4d26-824a-e859b4e4ac6b" (UID: "b3181951-f3a5-4d26-824a-e859b4e4ac6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:29.502259 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.502151 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-bundle" pod "330dc71b-1c41-488c-9024-4cc3fd9e46d3" (UID: "330dc71b-1c41-488c-9024-4cc3fd9e46d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:29.508253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.502615 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-bundle" pod "da33de69-4e86-485c-8c53-bf455b870bba" (UID: "da33de69-4e86-485c-8c53-bf455b870bba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:29.508253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.503249 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-bundle" pod "9393281e-e858-4be0-a5b9-af7159487b40" (UID: "9393281e-e858-4be0-a5b9-af7159487b40"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:29.508253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.507762 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-util" pod "330dc71b-1c41-488c-9024-4cc3fd9e46d3" (UID: "330dc71b-1c41-488c-9024-4cc3fd9e46d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:29.508253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.508026 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3181951-f3a5-4d26-824a-e859b4e4ac6b-kube-api-access-2hcmq" pod "b3181951-f3a5-4d26-824a-e859b4e4ac6b" (UID: "b3181951-f3a5-4d26-824a-e859b4e4ac6b"). InnerVolumeSpecName "kube-api-access-2hcmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:47:29.508959 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.508889 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-util" pod "da33de69-4e86-485c-8c53-bf455b870bba" (UID: "da33de69-4e86-485c-8c53-bf455b870bba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:29.510028 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.510003 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-util" pod "9393281e-e858-4be0-a5b9-af7159487b40" (UID: "9393281e-e858-4be0-a5b9-af7159487b40"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:29.510735 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.510715 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-util" pod "b3181951-f3a5-4d26-824a-e859b4e4ac6b" (UID: "b3181951-f3a5-4d26-824a-e859b4e4ac6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:47:29.511314 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.511292 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330dc71b-1c41-488c-9024-4cc3fd9e46d3-kube-api-access-n9rrr" pod "330dc71b-1c41-488c-9024-4cc3fd9e46d3" (UID: "330dc71b-1c41-488c-9024-4cc3fd9e46d3"). InnerVolumeSpecName "kube-api-access-n9rrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:47:29.511369 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.511295 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da33de69-4e86-485c-8c53-bf455b870bba-kube-api-access-g7bc2" pod "da33de69-4e86-485c-8c53-bf455b870bba" (UID: "da33de69-4e86-485c-8c53-bf455b870bba"). InnerVolumeSpecName "kube-api-access-g7bc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:47:29.512457 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.512435 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9393281e-e858-4be0-a5b9-af7159487b40-kube-api-access-dw5z8" pod "9393281e-e858-4be0-a5b9-af7159487b40" (UID: "9393281e-e858-4be0-a5b9-af7159487b40"). InnerVolumeSpecName "kube-api-access-dw5z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:47:29.601381 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601344 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601381 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601374 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-util\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601381 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601385 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hcmq\" (UniqueName: \"kubernetes.io/projected/b3181951-f3a5-4d26-824a-e859b4e4ac6b-kube-api-access-2hcmq\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601395 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-util\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601404 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9rrr\" (UniqueName: \"kubernetes.io/projected/330dc71b-1c41-488c-9024-4cc3fd9e46d3-kube-api-access-n9rrr\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601412 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7bc2\" (UniqueName: \"kubernetes.io/projected/da33de69-4e86-485c-8c53-bf455b870bba-kube-api-access-g7bc2\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601422 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw5z8\" (UniqueName: \"kubernetes.io/projected/9393281e-e858-4be0-a5b9-af7159487b40-kube-api-access-dw5z8\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601430 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/330dc71b-1c41-488c-9024-4cc3fd9e46d3-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601437 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9393281e-e858-4be0-a5b9-af7159487b40-util\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601445 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-util\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601452 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3181951-f3a5-4d26-824a-e859b4e4ac6b-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:29.601598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:29.601460 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da33de69-4e86-485c-8c53-bf455b870bba-bundle\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:47:30.193613 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.193533 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" Oct 29 21:47:30.193613 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.193540 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lw2c6" event={"ID":"b3181951-f3a5-4d26-824a-e859b4e4ac6b","Type":"ContainerDied","Data":"e0b7c5cc998f0e65c8ecae5ba9256f4c925b0cdda7350992976c9f7e48b0d23f"} Oct 29 21:47:30.193613 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.193592 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0b7c5cc998f0e65c8ecae5ba9256f4c925b0cdda7350992976c9f7e48b0d23f" Oct 29 21:47:30.195498 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.195470 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" event={"ID":"330dc71b-1c41-488c-9024-4cc3fd9e46d3","Type":"ContainerDied","Data":"d73e5ec96c5c949dcc794455c3d5c3513ce4ca4bf8d3d28ca1f57c112bb1ce99"} Oct 29 21:47:30.195498 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.195490 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nd6n4" Oct 29 21:47:30.195655 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.195510 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d73e5ec96c5c949dcc794455c3d5c3513ce4ca4bf8d3d28ca1f57c112bb1ce99" Oct 29 21:47:30.197395 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.197373 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" event={"ID":"da33de69-4e86-485c-8c53-bf455b870bba","Type":"ContainerDied","Data":"86ae982f172ea90205e0be5b8115ca7331a97d16fc8a59cfbccd2d609f5749c1"} Oct 29 21:47:30.197473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.197396 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxdz2" Oct 29 21:47:30.197473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.197411 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86ae982f172ea90205e0be5b8115ca7331a97d16fc8a59cfbccd2d609f5749c1" Oct 29 21:47:30.202655 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.202631 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" event={"ID":"9393281e-e858-4be0-a5b9-af7159487b40","Type":"ContainerDied","Data":"be3578b1e6ba989c05a918343a0adb0614581883f9dbfbfbe144b170f7bac413"} Oct 29 21:47:30.202731 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.202671 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be3578b1e6ba989c05a918343a0adb0614581883f9dbfbfbe144b170f7bac413" Oct 29 21:47:30.202731 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.202694 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dvmc9" Oct 29 21:47:30.840311 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:30.840277 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:31.021219 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:31.021189 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:33.819260 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:33.819220 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:36.443987 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:36.443950 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:36.621926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:36.621872 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:40.426368 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:40.426333 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:42.421551 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:42.421515 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:44.034132 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034098 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97"] Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034391 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3014a71b-1361-48cb-a132-4e1f0c2d6a3c" containerName="pull" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034402 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3014a71b-1361-48cb-a132-4e1f0c2d6a3c" containerName="pull" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034411 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3014a71b-1361-48cb-a132-4e1f0c2d6a3c" containerName="extract" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034416 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3014a71b-1361-48cb-a132-4e1f0c2d6a3c" containerName="extract" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034428 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c530300a-7545-46d0-b60f-0b3c04fa0b58" containerName="extract" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034433 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c530300a-7545-46d0-b60f-0b3c04fa0b58" containerName="extract" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034439 4952 cpu_manager.go:414] "containerMap: removing container" podUID="330dc71b-1c41-488c-9024-4cc3fd9e46d3" containerName="extract" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034443 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330dc71b-1c41-488c-9024-4cc3fd9e46d3" containerName="extract" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034467 4952 cpu_manager.go:414] "containerMap: removing container" podUID="330dc71b-1c41-488c-9024-4cc3fd9e46d3" containerName="pull" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034473 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330dc71b-1c41-488c-9024-4cc3fd9e46d3" containerName="pull" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034479 4952 cpu_manager.go:414] "containerMap: removing container" podUID="11433c8a-25be-4f64-a0d7-44422fd25a73" containerName="extract" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034484 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11433c8a-25be-4f64-a0d7-44422fd25a73" containerName="extract" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034496 4952 cpu_manager.go:414] "containerMap: removing container" podUID="11433c8a-25be-4f64-a0d7-44422fd25a73" containerName="util" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034501 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11433c8a-25be-4f64-a0d7-44422fd25a73" containerName="util" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034508 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3014a71b-1361-48cb-a132-4e1f0c2d6a3c" containerName="util" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034513 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3014a71b-1361-48cb-a132-4e1f0c2d6a3c" containerName="util" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034519 4952 cpu_manager.go:414] "containerMap: removing container" podUID="04b33341-64e3-4e16-b946-e192918745fa" containerName="pull" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034523 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b33341-64e3-4e16-b946-e192918745fa" containerName="pull" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034529 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9393281e-e858-4be0-a5b9-af7159487b40" containerName="pull" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034533 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9393281e-e858-4be0-a5b9-af7159487b40" containerName="pull" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034540 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b3181951-f3a5-4d26-824a-e859b4e4ac6b" containerName="pull" Oct 29 21:47:44.034536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034544 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3181951-f3a5-4d26-824a-e859b4e4ac6b" containerName="pull" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034552 4952 cpu_manager.go:414] "containerMap: removing container" podUID="da33de69-4e86-485c-8c53-bf455b870bba" containerName="pull" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034557 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da33de69-4e86-485c-8c53-bf455b870bba" containerName="pull" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034564 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c530300a-7545-46d0-b60f-0b3c04fa0b58" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034568 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c530300a-7545-46d0-b60f-0b3c04fa0b58" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034573 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c530300a-7545-46d0-b60f-0b3c04fa0b58" containerName="pull" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034577 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c530300a-7545-46d0-b60f-0b3c04fa0b58" containerName="pull" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034587 4952 cpu_manager.go:414] "containerMap: removing container" podUID="04b33341-64e3-4e16-b946-e192918745fa" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034592 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b33341-64e3-4e16-b946-e192918745fa" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034598 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b3181951-f3a5-4d26-824a-e859b4e4ac6b" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034602 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3181951-f3a5-4d26-824a-e859b4e4ac6b" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034608 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9393281e-e858-4be0-a5b9-af7159487b40" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034613 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9393281e-e858-4be0-a5b9-af7159487b40" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034617 4952 cpu_manager.go:414] "containerMap: removing container" podUID="04b33341-64e3-4e16-b946-e192918745fa" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034622 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b33341-64e3-4e16-b946-e192918745fa" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034629 4952 cpu_manager.go:414] "containerMap: removing container" podUID="da33de69-4e86-485c-8c53-bf455b870bba" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034634 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da33de69-4e86-485c-8c53-bf455b870bba" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034641 4952 cpu_manager.go:414] "containerMap: removing container" podUID="330dc71b-1c41-488c-9024-4cc3fd9e46d3" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034645 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330dc71b-1c41-488c-9024-4cc3fd9e46d3" containerName="util" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034650 4952 cpu_manager.go:414] "containerMap: removing container" podUID="da33de69-4e86-485c-8c53-bf455b870bba" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034654 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da33de69-4e86-485c-8c53-bf455b870bba" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034659 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9393281e-e858-4be0-a5b9-af7159487b40" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034663 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9393281e-e858-4be0-a5b9-af7159487b40" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034669 4952 cpu_manager.go:414] "containerMap: removing container" podUID="11433c8a-25be-4f64-a0d7-44422fd25a73" containerName="pull" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034674 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11433c8a-25be-4f64-a0d7-44422fd25a73" containerName="pull" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034679 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b3181951-f3a5-4d26-824a-e859b4e4ac6b" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034683 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3181951-f3a5-4d26-824a-e859b4e4ac6b" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034725 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="11433c8a-25be-4f64-a0d7-44422fd25a73" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034735 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b33341-64e3-4e16-b946-e192918745fa" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034741 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="da33de69-4e86-485c-8c53-bf455b870bba" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034748 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9393281e-e858-4be0-a5b9-af7159487b40" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034753 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3181951-f3a5-4d26-824a-e859b4e4ac6b" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034759 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3014a71b-1361-48cb-a132-4e1f0c2d6a3c" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034764 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c530300a-7545-46d0-b60f-0b3c04fa0b58" containerName="extract" Oct 29 21:47:44.035163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.034770 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="330dc71b-1c41-488c-9024-4cc3fd9e46d3" containerName="extract" Oct 29 21:47:44.037285 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.037268 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97" Oct 29 21:47:44.066790 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.066761 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 21:47:44.066790 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.066761 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 21:47:44.067043 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.066830 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-js627\"" Oct 29 21:47:44.068485 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.068461 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97"] Oct 29 21:47:44.219224 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.219143 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxznz\" (UniqueName: \"kubernetes.io/projected/df08f7e5-a162-4fb7-8896-4d77f606e4ce-kube-api-access-sxznz\") pod \"limitador-operator-controller-manager-5778bb4449-hfj97\" (UID: \"df08f7e5-a162-4fb7-8896-4d77f606e4ce\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97" Oct 29 21:47:44.319882 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.319796 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxznz\" (UniqueName: \"kubernetes.io/projected/df08f7e5-a162-4fb7-8896-4d77f606e4ce-kube-api-access-sxznz\") pod \"limitador-operator-controller-manager-5778bb4449-hfj97\" (UID: \"df08f7e5-a162-4fb7-8896-4d77f606e4ce\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97" Oct 29 21:47:44.348066 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.348034 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxznz\" (UniqueName: \"kubernetes.io/projected/df08f7e5-a162-4fb7-8896-4d77f606e4ce-kube-api-access-sxznz\") pod \"limitador-operator-controller-manager-5778bb4449-hfj97\" (UID: \"df08f7e5-a162-4fb7-8896-4d77f606e4ce\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97" Oct 29 21:47:44.348193 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.348151 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97" Oct 29 21:47:44.665704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.665677 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97"] Oct 29 21:47:44.668090 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:44.668064 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf08f7e5_a162_4fb7_8896_4d77f606e4ce.slice/crio-83bbb05b8c42f7f39d047de38d420a3b851e59da36841f862fc8ab58935b2d68 WatchSource:0}: Error finding container 83bbb05b8c42f7f39d047de38d420a3b851e59da36841f862fc8ab58935b2d68: Status 404 returned error can't find the container with id 83bbb05b8c42f7f39d047de38d420a3b851e59da36841f862fc8ab58935b2d68 Oct 29 21:47:44.670247 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.670232 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:47:44.832128 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:44.832073 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:45.267820 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:45.267781 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97" event={"ID":"df08f7e5-a162-4fb7-8896-4d77f606e4ce","Type":"ContainerStarted","Data":"83bbb05b8c42f7f39d047de38d420a3b851e59da36841f862fc8ab58935b2d68"} Oct 29 21:47:45.527310 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:47:45.527226 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:47:45.663231 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:45.663185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:46.453820 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.453783 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62"] Oct 29 21:47:46.462255 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.462234 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" Oct 29 21:47:46.473474 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.473417 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62"] Oct 29 21:47:46.481476 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.481452 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 29 21:47:46.481476 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.481473 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-frr7v\"" Oct 29 21:47:46.481668 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.481648 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 29 21:47:46.639154 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.639124 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a732686-8e7a-4152-83c9-f551a8ef6bc5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-92c62\" (UID: \"2a732686-8e7a-4152-83c9-f551a8ef6bc5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" Oct 29 21:47:46.639334 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.639166 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdfx6\" (UniqueName: \"kubernetes.io/projected/2a732686-8e7a-4152-83c9-f551a8ef6bc5-kube-api-access-cdfx6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-92c62\" (UID: \"2a732686-8e7a-4152-83c9-f551a8ef6bc5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" Oct 29 21:47:46.739871 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.739791 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdfx6\" (UniqueName: \"kubernetes.io/projected/2a732686-8e7a-4152-83c9-f551a8ef6bc5-kube-api-access-cdfx6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-92c62\" (UID: \"2a732686-8e7a-4152-83c9-f551a8ef6bc5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" Oct 29 21:47:46.740068 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.739927 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a732686-8e7a-4152-83c9-f551a8ef6bc5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-92c62\" (UID: \"2a732686-8e7a-4152-83c9-f551a8ef6bc5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" Oct 29 21:47:46.740314 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.740291 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a732686-8e7a-4152-83c9-f551a8ef6bc5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-92c62\" (UID: \"2a732686-8e7a-4152-83c9-f551a8ef6bc5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" Oct 29 21:47:46.770257 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.770228 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdfx6\" (UniqueName: \"kubernetes.io/projected/2a732686-8e7a-4152-83c9-f551a8ef6bc5-kube-api-access-cdfx6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-92c62\" (UID: \"2a732686-8e7a-4152-83c9-f551a8ef6bc5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" Oct 29 21:47:46.776931 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:46.776894 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" Oct 29 21:47:47.123865 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:47.123826 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62"] Oct 29 21:47:47.139705 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:47.139664 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a732686_8e7a_4152_83c9_f551a8ef6bc5.slice/crio-a7ce0f5c9a97cfa793e362a0af33c4864f0db80d5087ed3000aade232321a08e WatchSource:0}: Error finding container a7ce0f5c9a97cfa793e362a0af33c4864f0db80d5087ed3000aade232321a08e: Status 404 returned error can't find the container with id a7ce0f5c9a97cfa793e362a0af33c4864f0db80d5087ed3000aade232321a08e Oct 29 21:47:47.279241 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:47.279216 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" event={"ID":"2a732686-8e7a-4152-83c9-f551a8ef6bc5","Type":"ContainerStarted","Data":"a7ce0f5c9a97cfa793e362a0af33c4864f0db80d5087ed3000aade232321a08e"} Oct 29 21:47:48.289414 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:48.289377 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97" event={"ID":"df08f7e5-a162-4fb7-8896-4d77f606e4ce","Type":"ContainerStarted","Data":"93cfc6d2ba80f1cca36e9d68da8519aa02c3168475323f15ee8210c6b3edb6a9"} Oct 29 21:47:48.289888 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:48.289557 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97" Oct 29 21:47:48.333281 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:48.333222 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97" podStartSLOduration=1.7451256819999998 podStartE2EDuration="4.333204488s" totalImagesPullingTime="2.588078806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:48.330666303 +0000 UTC m=+333.805254722" watchObservedRunningTime="2026-10-29 21:47:48.333204488 +0000 UTC m=+333.807792906" Oct 29 21:47:49.042872 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.042784 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:49.075106 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.075077 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-khzn6"] Oct 29 21:47:49.078645 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.078625 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-khzn6" Oct 29 21:47:49.100697 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.100674 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rkqhq\"" Oct 29 21:47:49.109115 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.109089 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-khzn6"] Oct 29 21:47:49.232431 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.232397 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:49.263661 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.263619 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jmdt\" (UniqueName: \"kubernetes.io/projected/c071ca3b-2106-4523-a0a5-25f28f2a5a16-kube-api-access-9jmdt\") pod \"authorino-operator-7b7bb4bc77-khzn6\" (UID: \"c071ca3b-2106-4523-a0a5-25f28f2a5a16\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-khzn6" Oct 29 21:47:49.296055 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.295975 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" event={"ID":"2a732686-8e7a-4152-83c9-f551a8ef6bc5","Type":"ContainerStarted","Data":"cc7b9a34c052bfaf511abef470534b4b3969d4a38603a6b40a4fdb07e70e50b1"} Oct 29 21:47:49.364178 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.364143 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jmdt\" (UniqueName: \"kubernetes.io/projected/c071ca3b-2106-4523-a0a5-25f28f2a5a16-kube-api-access-9jmdt\") pod \"authorino-operator-7b7bb4bc77-khzn6\" (UID: \"c071ca3b-2106-4523-a0a5-25f28f2a5a16\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-khzn6" Oct 29 21:47:49.408279 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.408249 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jmdt\" (UniqueName: \"kubernetes.io/projected/c071ca3b-2106-4523-a0a5-25f28f2a5a16-kube-api-access-9jmdt\") pod \"authorino-operator-7b7bb4bc77-khzn6\" (UID: \"c071ca3b-2106-4523-a0a5-25f28f2a5a16\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-khzn6" Oct 29 21:47:49.527633 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.527597 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:49.691308 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:49.691275 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-khzn6" Oct 29 21:47:50.012082 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.012038 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-92c62" podStartSLOduration=2.56855265 podStartE2EDuration="4.012021107s" totalImagesPullingTime="1.443468457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:49.375114718 +0000 UTC m=+334.849703147" watchObservedRunningTime="2026-10-29 21:47:50.012021107 +0000 UTC m=+335.486609524" Oct 29 21:47:50.012805 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.012788 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-khzn6"] Oct 29 21:47:50.014768 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:50.014737 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc071ca3b_2106_4523_a0a5_25f28f2a5a16.slice/crio-e631e2aca06a0013256bdbc0149c77b0b60568a0a00e04c3478d97ec3e01ee63 WatchSource:0}: Error finding container e631e2aca06a0013256bdbc0149c77b0b60568a0a00e04c3478d97ec3e01ee63: Status 404 returned error can't find the container with id e631e2aca06a0013256bdbc0149c77b0b60568a0a00e04c3478d97ec3e01ee63 Oct 29 21:47:50.094288 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.094257 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:50.302807 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.302719 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-khzn6" event={"ID":"c071ca3b-2106-4523-a0a5-25f28f2a5a16","Type":"ContainerStarted","Data":"e631e2aca06a0013256bdbc0149c77b0b60568a0a00e04c3478d97ec3e01ee63"} Oct 29 21:47:50.468152 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.468117 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:50.617853 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.617236 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq"] Oct 29 21:47:50.620817 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.620796 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.640702 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.640678 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 21:47:50.640797 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.640752 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 29 21:47:50.652161 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.652140 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 29 21:47:50.652161 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.652152 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 21:47:50.652301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.652183 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-flksv\"" Oct 29 21:47:50.665504 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.665483 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq"] Oct 29 21:47:50.692245 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.692215 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:50.741329 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.741304 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 29 21:47:50.774217 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.774188 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/12b6333f-aa25-4a9d-bc86-380956056dd9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.774462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.774226 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/12b6333f-aa25-4a9d-bc86-380956056dd9-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.774462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.774256 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12b6333f-aa25-4a9d-bc86-380956056dd9-tmp\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.774462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.774338 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsq9d\" (UniqueName: \"kubernetes.io/projected/12b6333f-aa25-4a9d-bc86-380956056dd9-kube-api-access-zsq9d\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.774462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.774381 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/12b6333f-aa25-4a9d-bc86-380956056dd9-manifests\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.774650 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.774484 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12b6333f-aa25-4a9d-bc86-380956056dd9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.875372 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.875292 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/12b6333f-aa25-4a9d-bc86-380956056dd9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.875372 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.875335 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/12b6333f-aa25-4a9d-bc86-380956056dd9-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.875614 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.875458 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12b6333f-aa25-4a9d-bc86-380956056dd9-tmp\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.875614 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.875499 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsq9d\" (UniqueName: \"kubernetes.io/projected/12b6333f-aa25-4a9d-bc86-380956056dd9-kube-api-access-zsq9d\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.875614 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.875530 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/12b6333f-aa25-4a9d-bc86-380956056dd9-manifests\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.875614 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.875582 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12b6333f-aa25-4a9d-bc86-380956056dd9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.875979 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.875954 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/12b6333f-aa25-4a9d-bc86-380956056dd9-manifests\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.877704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.877680 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12b6333f-aa25-4a9d-bc86-380956056dd9-tmp\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.878025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.877998 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12b6333f-aa25-4a9d-bc86-380956056dd9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.878215 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.878195 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/12b6333f-aa25-4a9d-bc86-380956056dd9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.878215 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.878209 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/12b6333f-aa25-4a9d-bc86-380956056dd9-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.886464 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.886430 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:50.906605 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.906583 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsq9d\" (UniqueName: \"kubernetes.io/projected/12b6333f-aa25-4a9d-bc86-380956056dd9-kube-api-access-zsq9d\") pod \"opendatahub-operator-controller-manager-7bf7b6c69-d24cq\" (UID: \"12b6333f-aa25-4a9d-bc86-380956056dd9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:50.934048 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:50.934022 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:51.070174 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:51.070143 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:51.350534 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:51.350502 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq"] Oct 29 21:47:51.362281 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:51.362245 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:51.365629 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:51.365606 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b6333f_aa25_4a9d_bc86_380956056dd9.slice/crio-b5c22f9cb64ea847bbcfd8ca4b6b5cbdcdb72481aca79c3a09bb54342fbe39fc WatchSource:0}: Error finding container b5c22f9cb64ea847bbcfd8ca4b6b5cbdcdb72481aca79c3a09bb54342fbe39fc: Status 404 returned error can't find the container with id b5c22f9cb64ea847bbcfd8ca4b6b5cbdcdb72481aca79c3a09bb54342fbe39fc Oct 29 21:47:51.544523 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:51.544488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:52.221293 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:52.221256 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:52.314664 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:52.314623 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" event={"ID":"12b6333f-aa25-4a9d-bc86-380956056dd9","Type":"ContainerStarted","Data":"b5c22f9cb64ea847bbcfd8ca4b6b5cbdcdb72481aca79c3a09bb54342fbe39fc"} Oct 29 21:47:52.468668 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:52.468631 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:52.657760 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:52.657723 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:52.851449 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:52.851416 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:53.174621 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.174589 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:53.294321 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.294270 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd"] Oct 29 21:47:53.304707 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.304612 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd" Oct 29 21:47:53.321798 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.321761 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd"] Oct 29 21:47:53.330864 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.330841 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 29 21:47:53.331022 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.330892 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-68xqw\"" Oct 29 21:47:53.380081 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.380045 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:53.399289 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.399254 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcsqp\" (UniqueName: \"kubernetes.io/projected/e3de2a0e-ffae-4747-b6d8-03e2d601e991-kube-api-access-lcsqp\") pod \"dns-operator-controller-manager-8557cbdb84-qhdzd\" (UID: \"e3de2a0e-ffae-4747-b6d8-03e2d601e991\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd" Oct 29 21:47:53.500555 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.500469 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcsqp\" (UniqueName: \"kubernetes.io/projected/e3de2a0e-ffae-4747-b6d8-03e2d601e991-kube-api-access-lcsqp\") pod \"dns-operator-controller-manager-8557cbdb84-qhdzd\" (UID: \"e3de2a0e-ffae-4747-b6d8-03e2d601e991\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd" Oct 29 21:47:53.529961 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.529930 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcsqp\" (UniqueName: \"kubernetes.io/projected/e3de2a0e-ffae-4747-b6d8-03e2d601e991-kube-api-access-lcsqp\") pod \"dns-operator-controller-manager-8557cbdb84-qhdzd\" (UID: \"e3de2a0e-ffae-4747-b6d8-03e2d601e991\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd" Oct 29 21:47:53.618351 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.618311 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd" Oct 29 21:47:53.938226 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:53.938190 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd"] Oct 29 21:47:53.940379 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:47:53.940346 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3de2a0e_ffae_4747_b6d8_03e2d601e991.slice/crio-2b6c7bb2aa2d641427ce6453f42edfefe62345c9272b197616c370a062888ade WatchSource:0}: Error finding container 2b6c7bb2aa2d641427ce6453f42edfefe62345c9272b197616c370a062888ade: Status 404 returned error can't find the container with id 2b6c7bb2aa2d641427ce6453f42edfefe62345c9272b197616c370a062888ade Oct 29 21:47:54.233994 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:54.233913 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:54.327913 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:54.327871 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd" event={"ID":"e3de2a0e-ffae-4747-b6d8-03e2d601e991","Type":"ContainerStarted","Data":"2b6c7bb2aa2d641427ce6453f42edfefe62345c9272b197616c370a062888ade"} Oct 29 21:47:54.708989 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:54.708929 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:54.903685 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:54.903597 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:55.341480 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:55.341441 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-khzn6" event={"ID":"c071ca3b-2106-4523-a0a5-25f28f2a5a16","Type":"ContainerStarted","Data":"4c80131d94875d8fed6003ca3a0a15ce27678ada31d021e28e579eec8f122766"} Oct 29 21:47:55.341774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:55.341753 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-khzn6" Oct 29 21:47:55.344041 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:55.344015 4952 generic.go:356] "Generic (PLEG): container finished" podID="12b6333f-aa25-4a9d-bc86-380956056dd9" containerID="f5bfef53dd3d23ea82247bfb4a760e28bd12f38b5a9e66aefe603dc2fb06e76b" exitCode=0 Oct 29 21:47:55.344162 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:55.344090 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" event={"ID":"12b6333f-aa25-4a9d-bc86-380956056dd9","Type":"ContainerDied","Data":"f5bfef53dd3d23ea82247bfb4a760e28bd12f38b5a9e66aefe603dc2fb06e76b"} Oct 29 21:47:55.384617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:55.384571 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-khzn6" podStartSLOduration=1.787107896 podStartE2EDuration="6.384555719s" totalImagesPullingTime="4.597447823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:55.383361176 +0000 UTC m=+340.857949595" watchObservedRunningTime="2026-10-29 21:47:55.384555719 +0000 UTC m=+340.859144137" Oct 29 21:47:55.646617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:55.646588 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:56.168424 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:56.168344 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:56.351198 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:56.351143 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd" event={"ID":"e3de2a0e-ffae-4747-b6d8-03e2d601e991","Type":"ContainerStarted","Data":"5ab855da1f2c6080f7bceacbf0897c03e53ee657358451db1909644109228e03"} Oct 29 21:47:56.351473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:56.351335 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd" Oct 29 21:47:56.353483 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:56.353458 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" event={"ID":"12b6333f-aa25-4a9d-bc86-380956056dd9","Type":"ContainerStarted","Data":"dbb35b930a5ef61ce47e475bdea036fe062c83ac08ca3e90df75b174828cc526"} Oct 29 21:47:56.388459 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:56.388419 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:56.394069 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:56.394017 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd" podStartSLOduration=1.759847785 podStartE2EDuration="3.394002117s" totalImagesPullingTime="1.634154332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:56.39367619 +0000 UTC m=+341.868264608" watchObservedRunningTime="2026-10-29 21:47:56.394002117 +0000 UTC m=+341.868590536" Oct 29 21:47:56.461732 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:56.461624 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" podStartSLOduration=2.265263616 podStartE2EDuration="6.461611223s" totalImagesPullingTime="3.453896352s" totalInitContainerRuntime="742.451255ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:56.459646429 +0000 UTC m=+341.934234849" watchObservedRunningTime="2026-10-29 21:47:56.461611223 +0000 UTC m=+341.936199640" Oct 29 21:47:57.359153 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:57.359108 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:47:58.116482 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:58.116450 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:58.292458 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:58.292422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:47:59.298502 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:47:59.298467 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-hfj97" Oct 29 21:48:00.509183 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.509150 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h"] Oct 29 21:48:00.514489 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.514469 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 21:48:00.530243 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.530218 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h"] Oct 29 21:48:00.540411 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.540391 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q5plm\"" Oct 29 21:48:00.560737 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.560707 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68024384-a51d-40b9-bb47-d4e2baac27d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hj67h\" (UID: \"68024384-a51d-40b9-bb47-d4e2baac27d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 21:48:00.560845 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.560759 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vvv\" (UniqueName: \"kubernetes.io/projected/68024384-a51d-40b9-bb47-d4e2baac27d1-kube-api-access-q5vvv\") pod \"kuadrant-operator-controller-manager-678d87766f-hj67h\" (UID: \"68024384-a51d-40b9-bb47-d4e2baac27d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 21:48:00.661802 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.661769 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68024384-a51d-40b9-bb47-d4e2baac27d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hj67h\" (UID: \"68024384-a51d-40b9-bb47-d4e2baac27d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 21:48:00.661982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.661811 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vvv\" (UniqueName: \"kubernetes.io/projected/68024384-a51d-40b9-bb47-d4e2baac27d1-kube-api-access-q5vvv\") pod \"kuadrant-operator-controller-manager-678d87766f-hj67h\" (UID: \"68024384-a51d-40b9-bb47-d4e2baac27d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 21:48:00.662192 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.662171 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68024384-a51d-40b9-bb47-d4e2baac27d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hj67h\" (UID: \"68024384-a51d-40b9-bb47-d4e2baac27d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 21:48:00.687193 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.687165 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vvv\" (UniqueName: \"kubernetes.io/projected/68024384-a51d-40b9-bb47-d4e2baac27d1-kube-api-access-q5vvv\") pod \"kuadrant-operator-controller-manager-678d87766f-hj67h\" (UID: \"68024384-a51d-40b9-bb47-d4e2baac27d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 21:48:00.827748 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.827708 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 21:48:00.838693 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:00.838656 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:01.158708 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:01.158685 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h"] Oct 29 21:48:01.160863 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:48:01.160831 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68024384_a51d_40b9_bb47_d4e2baac27d1.slice/crio-c44405548ca5c5eaa62238c6f740cc4b1371974f4da8434e114eb42f208d0cc1 WatchSource:0}: Error finding container c44405548ca5c5eaa62238c6f740cc4b1371974f4da8434e114eb42f208d0cc1: Status 404 returned error can't find the container with id c44405548ca5c5eaa62238c6f740cc4b1371974f4da8434e114eb42f208d0cc1 Oct 29 21:48:01.378829 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:01.378794 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" event={"ID":"68024384-a51d-40b9-bb47-d4e2baac27d1","Type":"ContainerStarted","Data":"c44405548ca5c5eaa62238c6f740cc4b1371974f4da8434e114eb42f208d0cc1"} Oct 29 21:48:03.460785 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:03.460736 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:03.651504 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:03.651465 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:05.403032 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:05.402991 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" event={"ID":"68024384-a51d-40b9-bb47-d4e2baac27d1","Type":"ContainerStarted","Data":"ae0fc47802a892d86790e808c8275e8956b800e6f4ca5117fd9ca082d057a1b3"} Oct 29 21:48:05.403418 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:05.403108 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 21:48:05.473673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:05.473578 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" podStartSLOduration=1.426748863 podStartE2EDuration="5.473563541s" totalImagesPullingTime="4.046814678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:05.472212346 +0000 UTC m=+350.946800764" watchObservedRunningTime="2026-10-29 21:48:05.473563541 +0000 UTC m=+350.948151959" Oct 29 21:48:05.617797 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:05.617747 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:07.005396 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:07.005361 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:07.202972 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:07.202932 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:07.360566 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:07.360540 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-khzn6" Oct 29 21:48:07.360731 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:07.360611 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qhdzd" Oct 29 21:48:08.365757 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:08.365724 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bf7b6c69-d24cq" Oct 29 21:48:08.653866 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:08.653788 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:08.832525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:08.832489 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:09.238346 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:09.238313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:09.421435 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:09.421398 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:12.889380 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:12.889343 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:13.084460 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:13.084425 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:15.525186 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:48:15.525147 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:48:16.410609 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:16.410574 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 21:48:17.732715 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:17.732658 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:17.925510 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:17.925476 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:24.411284 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:24.411245 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:24.593020 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:24.592973 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:26.619305 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:26.619271 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:26.832628 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:26.832591 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:27.078651 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.078619 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:27.458361 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.458281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:27.641146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.641108 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:27.678173 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.678147 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925"] Oct 29 21:48:27.681670 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.681445 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.708049 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.708027 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dmrsl\"" Oct 29 21:48:27.716656 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.716603 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925"] Oct 29 21:48:27.786976 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.786949 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.787122 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.786980 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.787122 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.787003 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.787122 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.787019 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.787285 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.787191 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.787341 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.787283 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.787341 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.787318 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.787422 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.787356 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.787422 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.787404 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmq69\" (UniqueName: \"kubernetes.io/projected/94cbbe37-02d8-47d2-bc1f-264c9384afeb-kube-api-access-fmq69\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.826524 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.826489 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:27.888030 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.887999 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888030 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888032 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888255 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888053 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888255 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888170 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888255 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888232 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888368 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888324 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888368 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888356 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888391 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888419 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmq69\" (UniqueName: \"kubernetes.io/projected/94cbbe37-02d8-47d2-bc1f-264c9384afeb-kube-api-access-fmq69\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888464 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888742 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888720 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.888896 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888871 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.889053 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.888977 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.889267 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.889246 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.890695 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.890675 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.890953 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.890937 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.912529 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.912504 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cbbe37-02d8-47d2-bc1f-264c9384afeb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.912736 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.912715 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmq69\" (UniqueName: \"kubernetes.io/projected/94cbbe37-02d8-47d2-bc1f-264c9384afeb-kube-api-access-fmq69\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c5762925\" (UID: \"94cbbe37-02d8-47d2-bc1f-264c9384afeb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:27.996960 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:27.996863 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:28.016248 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.016215 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:28.322778 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.322754 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925"] Oct 29 21:48:28.324675 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:48:28.324646 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cbbe37_02d8_47d2_bc1f_264c9384afeb.slice/crio-d552261873ec4ffc54b9cfa03b1f2bb93a2b7ddf87ec22def13ad472c19fa063 WatchSource:0}: Error finding container d552261873ec4ffc54b9cfa03b1f2bb93a2b7ddf87ec22def13ad472c19fa063: Status 404 returned error can't find the container with id d552261873ec4ffc54b9cfa03b1f2bb93a2b7ddf87ec22def13ad472c19fa063 Oct 29 21:48:28.326742 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.326707 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:48:28.326847 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.326784 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:48:28.326924 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.326843 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:48:28.377293 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.377263 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:28.525181 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.525143 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" event={"ID":"94cbbe37-02d8-47d2-bc1f-264c9384afeb","Type":"ContainerStarted","Data":"84d0890cefa413f2291f0fa3607d322a491711732ced21b9dc4e44b86f22cc39"} Oct 29 21:48:28.525337 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.525218 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" event={"ID":"94cbbe37-02d8-47d2-bc1f-264c9384afeb","Type":"ContainerStarted","Data":"d552261873ec4ffc54b9cfa03b1f2bb93a2b7ddf87ec22def13ad472c19fa063"} Oct 29 21:48:28.548422 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.548385 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:28.574299 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.574252 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" podStartSLOduration=1.5742341579999999 podStartE2EDuration="1.574234158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:28.572793097 +0000 UTC m=+374.047381529" watchObservedRunningTime="2026-10-29 21:48:28.574234158 +0000 UTC m=+374.048822577" Oct 29 21:48:28.725610 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.725571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:28.928696 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.928630 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:28.997325 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:28.997295 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:29.137081 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.136996 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:29.501462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.501425 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:48:29.505045 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.505021 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" Oct 29 21:48:29.525790 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.525763 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:48:29.535038 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.535016 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jjjl6\"" Oct 29 21:48:29.535038 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.535022 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 29 21:48:29.562136 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.562115 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sg5gc"] Oct 29 21:48:29.567641 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.567591 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sg5gc" Oct 29 21:48:29.586736 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.586714 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sg5gc"] Oct 29 21:48:29.598849 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.598818 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:48:29.602272 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.602248 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cn7f\" (UniqueName: \"kubernetes.io/projected/dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63-kube-api-access-6cn7f\") pod \"limitador-limitador-7c9b87d574-7gmcc\" (UID: \"dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" Oct 29 21:48:29.602404 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.602385 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63-config-file\") pod \"limitador-limitador-7c9b87d574-7gmcc\" (UID: \"dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" Oct 29 21:48:29.623950 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.623926 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k7sdt\"" Oct 29 21:48:29.703809 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.703767 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63-config-file\") pod \"limitador-limitador-7c9b87d574-7gmcc\" (UID: \"dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" Oct 29 21:48:29.703809 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.703813 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twkp\" (UniqueName: \"kubernetes.io/projected/09274a49-e95e-4084-a7f3-236c9f9529b4-kube-api-access-7twkp\") pod \"authorino-57f7f8659f-sg5gc\" (UID: \"09274a49-e95e-4084-a7f3-236c9f9529b4\") " pod="kuadrant-system/authorino-57f7f8659f-sg5gc" Oct 29 21:48:29.704089 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.703849 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cn7f\" (UniqueName: \"kubernetes.io/projected/dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63-kube-api-access-6cn7f\") pod \"limitador-limitador-7c9b87d574-7gmcc\" (UID: \"dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" Oct 29 21:48:29.704514 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.704492 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63-config-file\") pod \"limitador-limitador-7c9b87d574-7gmcc\" (UID: \"dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" Oct 29 21:48:29.730539 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.730505 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:29.736432 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.736404 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cn7f\" (UniqueName: \"kubernetes.io/projected/dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63-kube-api-access-6cn7f\") pod \"limitador-limitador-7c9b87d574-7gmcc\" (UID: \"dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" Oct 29 21:48:29.805122 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.805056 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twkp\" (UniqueName: \"kubernetes.io/projected/09274a49-e95e-4084-a7f3-236c9f9529b4-kube-api-access-7twkp\") pod \"authorino-57f7f8659f-sg5gc\" (UID: \"09274a49-e95e-4084-a7f3-236c9f9529b4\") " pod="kuadrant-system/authorino-57f7f8659f-sg5gc" Oct 29 21:48:29.818091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.818072 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" Oct 29 21:48:29.830830 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.830808 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twkp\" (UniqueName: \"kubernetes.io/projected/09274a49-e95e-4084-a7f3-236c9f9529b4-kube-api-access-7twkp\") pod \"authorino-57f7f8659f-sg5gc\" (UID: \"09274a49-e95e-4084-a7f3-236c9f9529b4\") " pod="kuadrant-system/authorino-57f7f8659f-sg5gc" Oct 29 21:48:29.889177 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.889148 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sg5gc" Oct 29 21:48:29.908921 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:29.908867 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:30.001622 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:30.001591 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:30.117640 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:30.117617 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:48:30.119322 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:48:30.119295 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8e9b1d_d4ab_4ea4_96a6_7247aa4e2f63.slice/crio-c9e3e0c35d650e8b08847ce23208ba1352fd1d1c62304c98db6db91223db2918 WatchSource:0}: Error finding container c9e3e0c35d650e8b08847ce23208ba1352fd1d1c62304c98db6db91223db2918: Status 404 returned error can't find the container with id c9e3e0c35d650e8b08847ce23208ba1352fd1d1c62304c98db6db91223db2918 Oct 29 21:48:30.189290 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:30.189266 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sg5gc"] Oct 29 21:48:30.190897 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:48:30.190872 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09274a49_e95e_4084_a7f3_236c9f9529b4.slice/crio-23ea72ae585e41204502a922e0ab554c5a423232f9fcb807d5d91debccb7441e WatchSource:0}: Error finding container 23ea72ae585e41204502a922e0ab554c5a423232f9fcb807d5d91debccb7441e: Status 404 returned error can't find the container with id 23ea72ae585e41204502a922e0ab554c5a423232f9fcb807d5d91debccb7441e Oct 29 21:48:30.486362 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:30.486290 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:30.537280 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:30.537247 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sg5gc" event={"ID":"09274a49-e95e-4084-a7f3-236c9f9529b4","Type":"ContainerStarted","Data":"23ea72ae585e41204502a922e0ab554c5a423232f9fcb807d5d91debccb7441e"} Oct 29 21:48:30.538355 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:30.538330 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" event={"ID":"dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63","Type":"ContainerStarted","Data":"c9e3e0c35d650e8b08847ce23208ba1352fd1d1c62304c98db6db91223db2918"} Oct 29 21:48:30.538557 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:30.538541 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:30.539568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:30.539550 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c5762925" Oct 29 21:48:31.545289 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:31.545206 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" event={"ID":"dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63","Type":"ContainerStarted","Data":"e2154588ace584728904488857ab72d8495c2d327489af224baacbd3c0764d61"} Oct 29 21:48:31.588887 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:31.588835 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" podStartSLOduration=1.579649737 podStartE2EDuration="2.588819666s" totalImagesPullingTime="1.009169929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:31.586695331 +0000 UTC m=+377.061283761" watchObservedRunningTime="2026-10-29 21:48:31.588819666 +0000 UTC m=+377.063408083" Oct 29 21:48:32.187797 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:32.187700 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:32.366368 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:32.366333 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:32.549670 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:32.549579 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:32.552452 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:32.552422 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sg5gc" event={"ID":"09274a49-e95e-4084-a7f3-236c9f9529b4","Type":"ContainerStarted","Data":"0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e"} Oct 29 21:48:32.552629 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:32.552612 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" Oct 29 21:48:32.592525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:32.592483 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-sg5gc" podStartSLOduration=1.9021912429999999 podStartE2EDuration="3.592471635s" totalImagesPullingTime="1.690280392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:32.591218491 +0000 UTC m=+378.065806910" watchObservedRunningTime="2026-10-29 21:48:32.592471635 +0000 UTC m=+378.067060054" Oct 29 21:48:33.892028 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:33.891997 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sg5gc"] Oct 29 21:48:34.562433 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:34.562394 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-sg5gc" podUID="09274a49-e95e-4084-a7f3-236c9f9529b4" containerName="authorino" containerID="cri-o://0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e" gracePeriod=30 Oct 29 21:48:34.806222 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:34.806197 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sg5gc" Oct 29 21:48:34.950228 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:34.950195 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09274a49-e95e-4084-a7f3-236c9f9529b4-kube-api-access-7twkp\" (UniqueName: \"kubernetes.io/projected/09274a49-e95e-4084-a7f3-236c9f9529b4-kube-api-access-7twkp\") pod \"09274a49-e95e-4084-a7f3-236c9f9529b4\" (UID: \"09274a49-e95e-4084-a7f3-236c9f9529b4\") " Oct 29 21:48:34.952417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:34.952392 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09274a49-e95e-4084-a7f3-236c9f9529b4-kube-api-access-7twkp" pod "09274a49-e95e-4084-a7f3-236c9f9529b4" (UID: "09274a49-e95e-4084-a7f3-236c9f9529b4"). InnerVolumeSpecName "kube-api-access-7twkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:48:35.050922 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:35.050879 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7twkp\" (UniqueName: \"kubernetes.io/projected/09274a49-e95e-4084-a7f3-236c9f9529b4-kube-api-access-7twkp\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:48:35.569257 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:35.569224 4952 generic.go:356] "Generic (PLEG): container finished" podID="09274a49-e95e-4084-a7f3-236c9f9529b4" containerID="0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e" exitCode=0 Oct 29 21:48:35.569428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:35.569275 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sg5gc" Oct 29 21:48:35.569428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:35.569302 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sg5gc" event={"ID":"09274a49-e95e-4084-a7f3-236c9f9529b4","Type":"ContainerDied","Data":"0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e"} Oct 29 21:48:35.569428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:35.569350 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sg5gc" event={"ID":"09274a49-e95e-4084-a7f3-236c9f9529b4","Type":"ContainerDied","Data":"23ea72ae585e41204502a922e0ab554c5a423232f9fcb807d5d91debccb7441e"} Oct 29 21:48:35.569428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:35.569372 4952 scope.go:122] "RemoveContainer" containerID="0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e" Oct 29 21:48:35.579444 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:35.579427 4952 scope.go:122] "RemoveContainer" containerID="0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e" Oct 29 21:48:35.579739 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:48:35.579710 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e\": container with ID starting with 0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e not found: ID does not exist" containerID="0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e" Oct 29 21:48:35.579807 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:35.579741 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e"} err="failed to get container status \"0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e\": rpc error: code = NotFound desc = could not find container \"0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e\": container with ID starting with 0310782fb475700942e1f59085ccfc1fe996fa1588de9f3dfb8576fbe6c5357e not found: ID does not exist" Oct 29 21:48:35.626160 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:35.626131 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sg5gc"] Oct 29 21:48:35.631648 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:35.631627 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sg5gc"] Oct 29 21:48:36.641799 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:36.641764 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09274a49-e95e-4084-a7f3-236c9f9529b4" path="/var/lib/kubelet/pods/09274a49-e95e-4084-a7f3-236c9f9529b4/volumes" Oct 29 21:48:38.197200 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:38.197166 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:38.385105 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:38.385070 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:43.558861 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:43.558828 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-7gmcc" Oct 29 21:48:45.524969 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:48:45.524937 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:48:49.203998 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:49.203961 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:49.394237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:49.394201 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:49.583949 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:49.583890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:50.080627 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:50.080595 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:53.653396 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:53.653358 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:53.833195 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:53.833161 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:54.145262 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:54.145227 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:54.323092 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:54.323051 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:57.058021 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:57.057932 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:48:57.232647 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:48:57.232606 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:02.969050 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:02.969015 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:03.147282 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:03.147244 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:05.737343 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:05.737303 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:09.364457 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.364419 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:09.375808 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.375780 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ddb99595d-5qgsh"] Oct 29 21:49:09.376179 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.376166 4952 cpu_manager.go:414] "containerMap: removing container" podUID="09274a49-e95e-4084-a7f3-236c9f9529b4" containerName="authorino" Oct 29 21:49:09.376225 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.376180 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09274a49-e95e-4084-a7f3-236c9f9529b4" containerName="authorino" Oct 29 21:49:09.376265 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.376257 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="09274a49-e95e-4084-a7f3-236c9f9529b4" containerName="authorino" Oct 29 21:49:09.382170 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.382155 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.403137 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.403112 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 29 21:49:09.405240 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.405218 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddb99595d-5qgsh"] Oct 29 21:49:09.414719 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.414697 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 29 21:49:09.414824 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.414745 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 29 21:49:09.414824 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.414772 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 29 21:49:09.414824 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.414789 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 29 21:49:09.414950 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.414851 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qkgw8\"" Oct 29 21:49:09.443927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.443876 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-metrics-certs\") pod \"maas-controller-6ddb99595d-5qgsh\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.444094 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.443946 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-webhook-cert\") pod \"maas-controller-6ddb99595d-5qgsh\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.444094 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.444059 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt62j\" (UniqueName: \"kubernetes.io/projected/b3aa9658-e5e3-465d-8f7c-28febde39537-kube-api-access-rt62j\") pod \"maas-controller-6ddb99595d-5qgsh\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.544546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.544512 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt62j\" (UniqueName: \"kubernetes.io/projected/b3aa9658-e5e3-465d-8f7c-28febde39537-kube-api-access-rt62j\") pod \"maas-controller-6ddb99595d-5qgsh\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.544797 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.544613 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-metrics-certs\") pod \"maas-controller-6ddb99595d-5qgsh\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.544797 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.544653 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-webhook-cert\") pod \"maas-controller-6ddb99595d-5qgsh\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.545016 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.544972 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:09.547070 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.547048 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-webhook-cert\") pod \"maas-controller-6ddb99595d-5qgsh\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.547157 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.547103 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-metrics-certs\") pod \"maas-controller-6ddb99595d-5qgsh\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.569159 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.569134 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt62j\" (UniqueName: \"kubernetes.io/projected/b3aa9658-e5e3-465d-8f7c-28febde39537-kube-api-access-rt62j\") pod \"maas-controller-6ddb99595d-5qgsh\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.695756 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.695693 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:09.745198 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.744260 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:09.926922 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.926869 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d8bdd4cf6-g6mpt"] Oct 29 21:49:09.932704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.932680 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:09.941283 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:09.941261 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8bdd4cf6-g6mpt"] Oct 29 21:49:10.016858 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.016835 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddb99595d-5qgsh"] Oct 29 21:49:10.019083 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:49:10.019060 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3aa9658_e5e3_465d_8f7c_28febde39537.slice/crio-040eab4992f97754822338ab3784d33593eb3cc4a02cc9b4ce40a6dfdc642e47 WatchSource:0}: Error finding container 040eab4992f97754822338ab3784d33593eb3cc4a02cc9b4ce40a6dfdc642e47: Status 404 returned error can't find the container with id 040eab4992f97754822338ab3784d33593eb3cc4a02cc9b4ce40a6dfdc642e47 Oct 29 21:49:10.048879 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.048850 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f-webhook-cert\") pod \"maas-controller-6d8bdd4cf6-g6mpt\" (UID: \"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f\") " pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:10.048988 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.048893 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f-metrics-certs\") pod \"maas-controller-6d8bdd4cf6-g6mpt\" (UID: \"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f\") " pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:10.049033 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.048988 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgbx5\" (UniqueName: \"kubernetes.io/projected/bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f-kube-api-access-xgbx5\") pod \"maas-controller-6d8bdd4cf6-g6mpt\" (UID: \"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f\") " pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:10.150369 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.150330 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f-webhook-cert\") pod \"maas-controller-6d8bdd4cf6-g6mpt\" (UID: \"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f\") " pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:10.150549 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.150379 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f-metrics-certs\") pod \"maas-controller-6d8bdd4cf6-g6mpt\" (UID: \"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f\") " pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:10.150549 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.150503 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgbx5\" (UniqueName: \"kubernetes.io/projected/bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f-kube-api-access-xgbx5\") pod \"maas-controller-6d8bdd4cf6-g6mpt\" (UID: \"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f\") " pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:10.152875 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.152854 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f-webhook-cert\") pod \"maas-controller-6d8bdd4cf6-g6mpt\" (UID: \"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f\") " pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:10.153025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.152985 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f-metrics-certs\") pod \"maas-controller-6d8bdd4cf6-g6mpt\" (UID: \"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f\") " pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:10.179211 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.179181 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgbx5\" (UniqueName: \"kubernetes.io/projected/bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f-kube-api-access-xgbx5\") pod \"maas-controller-6d8bdd4cf6-g6mpt\" (UID: \"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f\") " pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:10.247874 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.247846 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:10.550801 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.550728 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8bdd4cf6-g6mpt"] Oct 29 21:49:10.555123 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:49:10.555089 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdefc3de_7cff_41cd_8cbc_d2dd14ef1e4f.slice/crio-d00110da55224ece55ed0f80178c46563e5d12333eac5963f94814b8073fab95 WatchSource:0}: Error finding container d00110da55224ece55ed0f80178c46563e5d12333eac5963f94814b8073fab95: Status 404 returned error can't find the container with id d00110da55224ece55ed0f80178c46563e5d12333eac5963f94814b8073fab95 Oct 29 21:49:10.746419 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.746383 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" event={"ID":"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f","Type":"ContainerStarted","Data":"d00110da55224ece55ed0f80178c46563e5d12333eac5963f94814b8073fab95"} Oct 29 21:49:10.747836 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:10.747802 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" event={"ID":"b3aa9658-e5e3-465d-8f7c-28febde39537","Type":"ContainerStarted","Data":"040eab4992f97754822338ab3784d33593eb3cc4a02cc9b4ce40a6dfdc642e47"} Oct 29 21:49:12.766203 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:12.766166 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" event={"ID":"b3aa9658-e5e3-465d-8f7c-28febde39537","Type":"ContainerStarted","Data":"3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4"} Oct 29 21:49:12.766577 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:12.766241 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:12.810853 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:12.810797 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" podStartSLOduration=1.136442897 podStartE2EDuration="3.810781905s" totalImagesPullingTime="2.674339008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:12.809220189 +0000 UTC m=+418.283808605" watchObservedRunningTime="2026-10-29 21:49:12.810781905 +0000 UTC m=+418.285370322" Oct 29 21:49:13.029398 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:13.029312 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:13.208317 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:13.208290 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:13.772642 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:13.772586 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" event={"ID":"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f","Type":"ContainerStarted","Data":"04fd5ba5f82b3b451af9a0e4ce24a35f893bd8e229e4d23902b75ee6e3b26223"} Oct 29 21:49:13.773129 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:13.772874 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:13.817508 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:13.817449 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" podStartSLOduration=2.240205801 podStartE2EDuration="4.817434673s" totalImagesPullingTime="2.577228872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:13.81527011 +0000 UTC m=+419.289858529" watchObservedRunningTime="2026-10-29 21:49:13.817434673 +0000 UTC m=+419.292023126" Oct 29 21:49:14.550438 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:14.550403 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:14.726171 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:14.726137 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:15.525023 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:49:15.524988 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:49:23.775536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:23.775498 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:24.779821 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:24.779789 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:49:24.869774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:24.869744 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ddb99595d-5qgsh"] Oct 29 21:49:24.870028 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:24.870001 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" podUID="b3aa9658-e5e3-465d-8f7c-28febde39537" containerName="manager" containerID="cri-o://3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4" gracePeriod=30 Oct 29 21:49:25.116004 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.115980 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:25.283228 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.283196 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-metrics-certs\") pod \"b3aa9658-e5e3-465d-8f7c-28febde39537\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " Oct 29 21:49:25.283228 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.283228 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-webhook-cert\") pod \"b3aa9658-e5e3-465d-8f7c-28febde39537\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " Oct 29 21:49:25.283433 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.283383 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3aa9658-e5e3-465d-8f7c-28febde39537-kube-api-access-rt62j\" (UniqueName: \"kubernetes.io/projected/b3aa9658-e5e3-465d-8f7c-28febde39537-kube-api-access-rt62j\") pod \"b3aa9658-e5e3-465d-8f7c-28febde39537\" (UID: \"b3aa9658-e5e3-465d-8f7c-28febde39537\") " Oct 29 21:49:25.285520 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.285489 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3aa9658-e5e3-465d-8f7c-28febde39537-kube-api-access-rt62j" pod "b3aa9658-e5e3-465d-8f7c-28febde39537" (UID: "b3aa9658-e5e3-465d-8f7c-28febde39537"). InnerVolumeSpecName "kube-api-access-rt62j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:49:25.285639 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.285551 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-webhook-cert" pod "b3aa9658-e5e3-465d-8f7c-28febde39537" (UID: "b3aa9658-e5e3-465d-8f7c-28febde39537"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:49:25.285639 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.285582 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-metrics-certs" pod "b3aa9658-e5e3-465d-8f7c-28febde39537" (UID: "b3aa9658-e5e3-465d-8f7c-28febde39537"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:49:25.384865 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.384820 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt62j\" (UniqueName: \"kubernetes.io/projected/b3aa9658-e5e3-465d-8f7c-28febde39537-kube-api-access-rt62j\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:49:25.384865 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.384861 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:49:25.384865 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.384870 4952 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3aa9658-e5e3-465d-8f7c-28febde39537-webhook-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:49:25.844493 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.844458 4952 generic.go:356] "Generic (PLEG): container finished" podID="b3aa9658-e5e3-465d-8f7c-28febde39537" containerID="3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4" exitCode=0 Oct 29 21:49:25.844943 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.844512 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" Oct 29 21:49:25.844943 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.844549 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" event={"ID":"b3aa9658-e5e3-465d-8f7c-28febde39537","Type":"ContainerDied","Data":"3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4"} Oct 29 21:49:25.844943 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.844604 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddb99595d-5qgsh" event={"ID":"b3aa9658-e5e3-465d-8f7c-28febde39537","Type":"ContainerDied","Data":"040eab4992f97754822338ab3784d33593eb3cc4a02cc9b4ce40a6dfdc642e47"} Oct 29 21:49:25.844943 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.844625 4952 scope.go:122] "RemoveContainer" containerID="3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4" Oct 29 21:49:25.853566 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.853550 4952 scope.go:122] "RemoveContainer" containerID="3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4" Oct 29 21:49:25.853889 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:49:25.853870 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4\": container with ID starting with 3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4 not found: ID does not exist" containerID="3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4" Oct 29 21:49:25.853984 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.853924 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4"} err="failed to get container status \"3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4\": rpc error: code = NotFound desc = could not find container \"3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4\": container with ID starting with 3ec09adaf735ae76f86e9cd8e33acef563585c7a97bc71c900c86da85a9735d4 not found: ID does not exist" Oct 29 21:49:25.897875 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.897853 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ddb99595d-5qgsh"] Oct 29 21:49:25.910111 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:25.910083 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6ddb99595d-5qgsh"] Oct 29 21:49:26.642458 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:26.642426 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3aa9658-e5e3-465d-8f7c-28febde39537" path="/var/lib/kubelet/pods/b3aa9658-e5e3-465d-8f7c-28febde39537/volumes" Oct 29 21:49:28.398455 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.398417 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:28.637391 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.637349 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:28.650641 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.650575 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hwxzn"] Oct 29 21:49:28.650964 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.650950 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b3aa9658-e5e3-465d-8f7c-28febde39537" containerName="manager" Oct 29 21:49:28.651007 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.650965 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3aa9658-e5e3-465d-8f7c-28febde39537" containerName="manager" Oct 29 21:49:28.651045 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.651032 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3aa9658-e5e3-465d-8f7c-28febde39537" containerName="manager" Oct 29 21:49:28.654002 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.653984 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" Oct 29 21:49:28.672526 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.672505 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-njtm5\"" Oct 29 21:49:28.674159 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.674135 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hwxzn"] Oct 29 21:49:28.683883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.683855 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 29 21:49:28.691752 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.691728 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk"] Oct 29 21:49:28.695248 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.695232 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:49:28.721711 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.721677 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk"] Oct 29 21:49:28.743857 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.743818 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pkknp"] Oct 29 21:49:28.749132 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.749110 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" Oct 29 21:49:28.768049 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.768021 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pkknp"] Oct 29 21:49:28.773129 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.773094 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 29 21:49:28.773261 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.773130 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6p4sx\"" Oct 29 21:49:28.773261 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.773225 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 29 21:49:28.773340 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.773290 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-xq79z\"" Oct 29 21:49:28.787374 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.787351 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-bk2cb"] Oct 29 21:49:28.790869 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.790847 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:28.812082 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.812057 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bk2cb"] Oct 29 21:49:28.816712 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.816682 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rq4\" (UniqueName: \"kubernetes.io/projected/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-kube-api-access-b4rq4\") pod \"llmisvc-controller-manager-869d5dcc5d-b6vdk\" (UID: \"9e2d99fc-781e-4b37-a85a-a7110cde7c8b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:49:28.816837 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.816733 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-b6vdk\" (UID: \"9e2d99fc-781e-4b37-a85a-a7110cde7c8b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:49:28.816875 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.816850 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn74x\" (UniqueName: \"kubernetes.io/projected/1ea74cd9-058e-4f28-b2b5-b64902ca8d75-kube-api-access-tn74x\") pod \"kserve-controller-manager-776cc66d47-hwxzn\" (UID: \"1ea74cd9-058e-4f28-b2b5-b64902ca8d75\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" Oct 29 21:49:28.816935 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.816924 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ea74cd9-058e-4f28-b2b5-b64902ca8d75-cert\") pod \"kserve-controller-manager-776cc66d47-hwxzn\" (UID: \"1ea74cd9-058e-4f28-b2b5-b64902ca8d75\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" Oct 29 21:49:28.823130 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.823107 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:28.873594 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.873563 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-424rm\"" Oct 29 21:49:28.873757 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.873616 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 29 21:49:28.873757 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.873632 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 29 21:49:28.918147 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.918071 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn74x\" (UniqueName: \"kubernetes.io/projected/1ea74cd9-058e-4f28-b2b5-b64902ca8d75-kube-api-access-tn74x\") pod \"kserve-controller-manager-776cc66d47-hwxzn\" (UID: \"1ea74cd9-058e-4f28-b2b5-b64902ca8d75\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" Oct 29 21:49:28.918147 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.918109 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc442ee-a458-43af-84a6-51d7fa97c5bd-tls-certs\") pod \"model-serving-api-75bd89fdfc-pkknp\" (UID: \"4cc442ee-a458-43af-84a6-51d7fa97c5bd\") " pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" Oct 29 21:49:28.918147 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.918129 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqwl9\" (UniqueName: \"kubernetes.io/projected/f33f40ca-9443-4c79-a820-3c5fbad96529-kube-api-access-sqwl9\") pod \"odh-model-controller-769875db56-bk2cb\" (UID: \"f33f40ca-9443-4c79-a820-3c5fbad96529\") " pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:28.918346 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.918206 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ea74cd9-058e-4f28-b2b5-b64902ca8d75-cert\") pod \"kserve-controller-manager-776cc66d47-hwxzn\" (UID: \"1ea74cd9-058e-4f28-b2b5-b64902ca8d75\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" Oct 29 21:49:28.918346 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.918258 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f33f40ca-9443-4c79-a820-3c5fbad96529-cert\") pod \"odh-model-controller-769875db56-bk2cb\" (UID: \"f33f40ca-9443-4c79-a820-3c5fbad96529\") " pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:28.918346 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.918292 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbm7x\" (UniqueName: \"kubernetes.io/projected/4cc442ee-a458-43af-84a6-51d7fa97c5bd-kube-api-access-zbm7x\") pod \"model-serving-api-75bd89fdfc-pkknp\" (UID: \"4cc442ee-a458-43af-84a6-51d7fa97c5bd\") " pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" Oct 29 21:49:28.918346 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.918338 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f33f40ca-9443-4c79-a820-3c5fbad96529-metrics-certs\") pod \"odh-model-controller-769875db56-bk2cb\" (UID: \"f33f40ca-9443-4c79-a820-3c5fbad96529\") " pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:28.918498 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.918389 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rq4\" (UniqueName: \"kubernetes.io/projected/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-kube-api-access-b4rq4\") pod \"llmisvc-controller-manager-869d5dcc5d-b6vdk\" (UID: \"9e2d99fc-781e-4b37-a85a-a7110cde7c8b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:49:28.918498 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.918431 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-b6vdk\" (UID: \"9e2d99fc-781e-4b37-a85a-a7110cde7c8b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:49:28.920858 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.920829 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ea74cd9-058e-4f28-b2b5-b64902ca8d75-cert\") pod \"kserve-controller-manager-776cc66d47-hwxzn\" (UID: \"1ea74cd9-058e-4f28-b2b5-b64902ca8d75\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" Oct 29 21:49:28.920975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.920842 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-b6vdk\" (UID: \"9e2d99fc-781e-4b37-a85a-a7110cde7c8b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:49:28.943417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.943388 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn74x\" (UniqueName: \"kubernetes.io/projected/1ea74cd9-058e-4f28-b2b5-b64902ca8d75-kube-api-access-tn74x\") pod \"kserve-controller-manager-776cc66d47-hwxzn\" (UID: \"1ea74cd9-058e-4f28-b2b5-b64902ca8d75\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" Oct 29 21:49:28.943556 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.943473 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rq4\" (UniqueName: \"kubernetes.io/projected/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-kube-api-access-b4rq4\") pod \"llmisvc-controller-manager-869d5dcc5d-b6vdk\" (UID: \"9e2d99fc-781e-4b37-a85a-a7110cde7c8b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:49:28.967050 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:28.967025 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" Oct 29 21:49:29.008595 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.008562 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:49:29.019941 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.019914 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f33f40ca-9443-4c79-a820-3c5fbad96529-cert\") pod \"odh-model-controller-769875db56-bk2cb\" (UID: \"f33f40ca-9443-4c79-a820-3c5fbad96529\") " pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:29.020012 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.019967 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbm7x\" (UniqueName: \"kubernetes.io/projected/4cc442ee-a458-43af-84a6-51d7fa97c5bd-kube-api-access-zbm7x\") pod \"model-serving-api-75bd89fdfc-pkknp\" (UID: \"4cc442ee-a458-43af-84a6-51d7fa97c5bd\") " pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" Oct 29 21:49:29.020049 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.020016 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f33f40ca-9443-4c79-a820-3c5fbad96529-metrics-certs\") pod \"odh-model-controller-769875db56-bk2cb\" (UID: \"f33f40ca-9443-4c79-a820-3c5fbad96529\") " pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:29.020127 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.020104 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc442ee-a458-43af-84a6-51d7fa97c5bd-tls-certs\") pod \"model-serving-api-75bd89fdfc-pkknp\" (UID: \"4cc442ee-a458-43af-84a6-51d7fa97c5bd\") " pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" Oct 29 21:49:29.020331 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.020135 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqwl9\" (UniqueName: \"kubernetes.io/projected/f33f40ca-9443-4c79-a820-3c5fbad96529-kube-api-access-sqwl9\") pod \"odh-model-controller-769875db56-bk2cb\" (UID: \"f33f40ca-9443-4c79-a820-3c5fbad96529\") " pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:29.024832 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.023600 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc442ee-a458-43af-84a6-51d7fa97c5bd-tls-certs\") pod \"model-serving-api-75bd89fdfc-pkknp\" (UID: \"4cc442ee-a458-43af-84a6-51d7fa97c5bd\") " pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" Oct 29 21:49:29.024832 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.023780 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f33f40ca-9443-4c79-a820-3c5fbad96529-metrics-certs\") pod \"odh-model-controller-769875db56-bk2cb\" (UID: \"f33f40ca-9443-4c79-a820-3c5fbad96529\") " pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:29.024832 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.024170 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f33f40ca-9443-4c79-a820-3c5fbad96529-cert\") pod \"odh-model-controller-769875db56-bk2cb\" (UID: \"f33f40ca-9443-4c79-a820-3c5fbad96529\") " pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:29.047351 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.047286 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbm7x\" (UniqueName: \"kubernetes.io/projected/4cc442ee-a458-43af-84a6-51d7fa97c5bd-kube-api-access-zbm7x\") pod \"model-serving-api-75bd89fdfc-pkknp\" (UID: \"4cc442ee-a458-43af-84a6-51d7fa97c5bd\") " pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" Oct 29 21:49:29.049979 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.047691 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqwl9\" (UniqueName: \"kubernetes.io/projected/f33f40ca-9443-4c79-a820-3c5fbad96529-kube-api-access-sqwl9\") pod \"odh-model-controller-769875db56-bk2cb\" (UID: \"f33f40ca-9443-4c79-a820-3c5fbad96529\") " pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:29.062434 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.062404 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" Oct 29 21:49:29.103135 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.103104 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:29.238364 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.238312 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:29.274766 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.274741 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hwxzn"] Oct 29 21:49:29.276411 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:49:29.276379 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea74cd9_058e_4f28_b2b5_b64902ca8d75.slice/crio-34dd30589547f54a4adf3a59db640a34e4f7c99c312c04ac477fe62cb7861d0d WatchSource:0}: Error finding container 34dd30589547f54a4adf3a59db640a34e4f7c99c312c04ac477fe62cb7861d0d: Status 404 returned error can't find the container with id 34dd30589547f54a4adf3a59db640a34e4f7c99c312c04ac477fe62cb7861d0d Oct 29 21:49:29.316293 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.316272 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk"] Oct 29 21:49:29.317736 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:49:29.317709 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9e2d99fc_781e_4b37_a85a_a7110cde7c8b.slice/crio-add7481f4a3e2b930527a54b6efb23aef1d3de6f9cf41a83579e433880d3a47b WatchSource:0}: Error finding container add7481f4a3e2b930527a54b6efb23aef1d3de6f9cf41a83579e433880d3a47b: Status 404 returned error can't find the container with id add7481f4a3e2b930527a54b6efb23aef1d3de6f9cf41a83579e433880d3a47b Oct 29 21:49:29.377577 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.377557 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pkknp"] Oct 29 21:49:29.379086 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:49:29.379060 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc442ee_a458_43af_84a6_51d7fa97c5bd.slice/crio-f93cfc1ef4e0e61a112947ac97a0fc39a6ef9e2821887dc3ada758cae1e592d9 WatchSource:0}: Error finding container f93cfc1ef4e0e61a112947ac97a0fc39a6ef9e2821887dc3ada758cae1e592d9: Status 404 returned error can't find the container with id f93cfc1ef4e0e61a112947ac97a0fc39a6ef9e2821887dc3ada758cae1e592d9 Oct 29 21:49:29.420761 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.420730 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bk2cb"] Oct 29 21:49:29.422057 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:49:29.422034 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33f40ca_9443_4c79_a820_3c5fbad96529.slice/crio-aedc9487168973c981770c26d831a55094e4fb08210fb6c7854d58e004ad035f WatchSource:0}: Error finding container aedc9487168973c981770c26d831a55094e4fb08210fb6c7854d58e004ad035f: Status 404 returned error can't find the container with id aedc9487168973c981770c26d831a55094e4fb08210fb6c7854d58e004ad035f Oct 29 21:49:29.872973 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.872929 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bk2cb" event={"ID":"f33f40ca-9443-4c79-a820-3c5fbad96529","Type":"ContainerStarted","Data":"aedc9487168973c981770c26d831a55094e4fb08210fb6c7854d58e004ad035f"} Oct 29 21:49:29.874504 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.874471 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" event={"ID":"1ea74cd9-058e-4f28-b2b5-b64902ca8d75","Type":"ContainerStarted","Data":"34dd30589547f54a4adf3a59db640a34e4f7c99c312c04ac477fe62cb7861d0d"} Oct 29 21:49:29.876768 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.876744 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" event={"ID":"9e2d99fc-781e-4b37-a85a-a7110cde7c8b","Type":"ContainerStarted","Data":"add7481f4a3e2b930527a54b6efb23aef1d3de6f9cf41a83579e433880d3a47b"} Oct 29 21:49:29.878351 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:29.878329 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" event={"ID":"4cc442ee-a458-43af-84a6-51d7fa97c5bd","Type":"ContainerStarted","Data":"f93cfc1ef4e0e61a112947ac97a0fc39a6ef9e2821887dc3ada758cae1e592d9"} Oct 29 21:49:31.233305 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:31.233264 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:31.422429 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:31.422394 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:31.613410 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:31.613237 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:31.838932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:31.838872 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:32.904879 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:32.904840 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" event={"ID":"4cc442ee-a458-43af-84a6-51d7fa97c5bd","Type":"ContainerStarted","Data":"6c6542a0ff3d656d5ec6902c572a3164e48b2229601e30724d1610da75b2729d"} Oct 29 21:49:32.905795 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:32.905769 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" Oct 29 21:49:32.969509 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:32.969444 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" podStartSLOduration=1.812506032 podStartE2EDuration="4.969424038s" totalImagesPullingTime="3.156918006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:32.966268115 +0000 UTC m=+438.440856560" watchObservedRunningTime="2026-10-29 21:49:32.969424038 +0000 UTC m=+438.444012458" Oct 29 21:49:33.916192 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:33.916151 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bk2cb" event={"ID":"f33f40ca-9443-4c79-a820-3c5fbad96529","Type":"ContainerStarted","Data":"8814bb2f649e94a7a22a7c64d7340f403f5393a7d8f5e52d5b334a5273626fea"} Oct 29 21:49:33.916612 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:33.916376 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:33.917767 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:33.917741 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" event={"ID":"1ea74cd9-058e-4f28-b2b5-b64902ca8d75","Type":"ContainerStarted","Data":"a73ebb0815dd6659bf272947d5caadefd4ea7825aebf8aa5c0805e83caa13d54"} Oct 29 21:49:33.917934 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:33.917821 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" Oct 29 21:49:33.919342 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:33.919315 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" event={"ID":"9e2d99fc-781e-4b37-a85a-a7110cde7c8b","Type":"ContainerStarted","Data":"24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0"} Oct 29 21:49:33.919543 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:33.919527 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:49:33.959629 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:33.959576 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-bk2cb" podStartSLOduration=1.837553422 podStartE2EDuration="5.959557453s" totalImagesPullingTime="4.122004031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:33.958285662 +0000 UTC m=+439.432874078" watchObservedRunningTime="2026-10-29 21:49:33.959557453 +0000 UTC m=+439.434145869" Oct 29 21:49:34.003332 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:34.003285 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" podStartSLOduration=1.6777987859999999 podStartE2EDuration="6.003271703s" totalImagesPullingTime="4.325472917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:34.000982974 +0000 UTC m=+439.475571393" watchObservedRunningTime="2026-10-29 21:49:34.003271703 +0000 UTC m=+439.477860112" Oct 29 21:49:34.049692 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:34.049596 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" podStartSLOduration=1.770969729 podStartE2EDuration="6.049576195s" totalImagesPullingTime="4.278606466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:34.046376912 +0000 UTC m=+439.520965330" watchObservedRunningTime="2026-10-29 21:49:34.049576195 +0000 UTC m=+439.524164615" Oct 29 21:49:43.692263 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:43.692231 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:43.885736 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:43.885686 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:44.927044 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:44.927010 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-bk2cb" Oct 29 21:49:44.928953 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:44.928933 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-pkknp" Oct 29 21:49:45.136301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:45.136248 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:45.347414 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:45.347377 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:45.525938 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:49:45.525888 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:49:45.534138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:45.534111 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:47.379089 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:47.379057 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:49:55.721668 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:55.721625 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:55.975172 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:55.975074 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:49:56.019261 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.019234 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qc9gq"] Oct 29 21:49:56.021633 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.021616 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" Oct 29 21:49:56.043461 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.043427 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq9tb\" (UniqueName: \"kubernetes.io/projected/ba84b25a-dfd9-4cb8-9316-f29128e39102-kube-api-access-pq9tb\") pod \"authorino-6f6944bcdd-qc9gq\" (UID: \"ba84b25a-dfd9-4cb8-9316-f29128e39102\") " pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" Oct 29 21:49:56.053525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.053508 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k7sdt\"" Oct 29 21:49:56.059930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.059894 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qc9gq"] Oct 29 21:49:56.062982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.062953 4952 generic.go:356] "Generic (PLEG): container finished" podID="ebc99afd-f631-46a3-8d94-bfaae865a10a" containerID="1f4fe74fd10c390eeaa683387f563eea6b1ff82441ba5b09f77326e67058b923" exitCode=0 Oct 29 21:49:56.063080 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.063027 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" event={"ID":"ebc99afd-f631-46a3-8d94-bfaae865a10a","Type":"ContainerDied","Data":"1f4fe74fd10c390eeaa683387f563eea6b1ff82441ba5b09f77326e67058b923"} Oct 29 21:49:56.063428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.063415 4952 scope.go:122] "RemoveContainer" containerID="1f4fe74fd10c390eeaa683387f563eea6b1ff82441ba5b09f77326e67058b923" Oct 29 21:49:56.064536 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.064506 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:49:56.064603 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.064566 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:49:56.064603 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.064596 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:49:56.143941 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.143897 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq9tb\" (UniqueName: \"kubernetes.io/projected/ba84b25a-dfd9-4cb8-9316-f29128e39102-kube-api-access-pq9tb\") pod \"authorino-6f6944bcdd-qc9gq\" (UID: \"ba84b25a-dfd9-4cb8-9316-f29128e39102\") " pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" Oct 29 21:49:56.174955 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.174869 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq9tb\" (UniqueName: \"kubernetes.io/projected/ba84b25a-dfd9-4cb8-9316-f29128e39102-kube-api-access-pq9tb\") pod \"authorino-6f6944bcdd-qc9gq\" (UID: \"ba84b25a-dfd9-4cb8-9316-f29128e39102\") " pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" Oct 29 21:49:56.274641 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.274553 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qc9gq"] Oct 29 21:49:56.274822 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.274805 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" Oct 29 21:49:56.318837 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.318801 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-zbqld"] Oct 29 21:49:56.322684 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.322658 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-zbqld" Oct 29 21:49:56.353486 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.341892 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 29 21:49:56.353486 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.344322 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-zbqld"] Oct 29 21:49:56.354179 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.353885 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfxb\" (UniqueName: \"kubernetes.io/projected/64362e63-6460-4e1c-9827-d83ef2c0b596-kube-api-access-wkfxb\") pod \"authorino-6889cbd8c4-zbqld\" (UID: \"64362e63-6460-4e1c-9827-d83ef2c0b596\") " pod="kuadrant-system/authorino-6889cbd8c4-zbqld" Oct 29 21:49:56.354179 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.353948 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64362e63-6460-4e1c-9827-d83ef2c0b596-tls-cert\") pod \"authorino-6889cbd8c4-zbqld\" (UID: \"64362e63-6460-4e1c-9827-d83ef2c0b596\") " pod="kuadrant-system/authorino-6889cbd8c4-zbqld" Oct 29 21:49:56.374922 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.374876 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-zbqld"] Oct 29 21:49:56.375288 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:49:56.375261 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wkfxb tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-zbqld" podUID="64362e63-6460-4e1c-9827-d83ef2c0b596" Oct 29 21:49:56.455107 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.455075 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfxb\" (UniqueName: \"kubernetes.io/projected/64362e63-6460-4e1c-9827-d83ef2c0b596-kube-api-access-wkfxb\") pod \"authorino-6889cbd8c4-zbqld\" (UID: \"64362e63-6460-4e1c-9827-d83ef2c0b596\") " pod="kuadrant-system/authorino-6889cbd8c4-zbqld" Oct 29 21:49:56.455107 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.455111 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64362e63-6460-4e1c-9827-d83ef2c0b596-tls-cert\") pod \"authorino-6889cbd8c4-zbqld\" (UID: \"64362e63-6460-4e1c-9827-d83ef2c0b596\") " pod="kuadrant-system/authorino-6889cbd8c4-zbqld" Oct 29 21:49:56.457734 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.457704 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64362e63-6460-4e1c-9827-d83ef2c0b596-tls-cert\") pod \"authorino-6889cbd8c4-zbqld\" (UID: \"64362e63-6460-4e1c-9827-d83ef2c0b596\") " pod="kuadrant-system/authorino-6889cbd8c4-zbqld" Oct 29 21:49:56.482336 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.482298 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfxb\" (UniqueName: \"kubernetes.io/projected/64362e63-6460-4e1c-9827-d83ef2c0b596-kube-api-access-wkfxb\") pod \"authorino-6889cbd8c4-zbqld\" (UID: \"64362e63-6460-4e1c-9827-d83ef2c0b596\") " pod="kuadrant-system/authorino-6889cbd8c4-zbqld" Oct 29 21:49:56.589617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:56.589587 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qc9gq"] Oct 29 21:49:56.592032 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:49:56.592007 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba84b25a_dfd9_4cb8_9316_f29128e39102.slice/crio-1bbe495987a7970eba0310ee5d76169585118b710c24ebff91fe8fb8ce83d1b4 WatchSource:0}: Error finding container 1bbe495987a7970eba0310ee5d76169585118b710c24ebff91fe8fb8ce83d1b4: Status 404 returned error can't find the container with id 1bbe495987a7970eba0310ee5d76169585118b710c24ebff91fe8fb8ce83d1b4 Oct 29 21:49:57.070023 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.069932 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" event={"ID":"ba84b25a-dfd9-4cb8-9316-f29128e39102","Type":"ContainerStarted","Data":"1bbe495987a7970eba0310ee5d76169585118b710c24ebff91fe8fb8ce83d1b4"} Oct 29 21:49:57.071993 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.071967 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" event={"ID":"ebc99afd-f631-46a3-8d94-bfaae865a10a","Type":"ContainerStarted","Data":"5eadf9d6a6121372c183ced5a2f37e080fe115197b97a3061b4c6148ec10e703"} Oct 29 21:49:57.072102 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.072008 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-zbqld" Oct 29 21:49:57.078700 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.078677 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-zbqld" Oct 29 21:49:57.159295 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.159256 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64362e63-6460-4e1c-9827-d83ef2c0b596-kube-api-access-wkfxb\" (UniqueName: \"kubernetes.io/projected/64362e63-6460-4e1c-9827-d83ef2c0b596-kube-api-access-wkfxb\") pod \"64362e63-6460-4e1c-9827-d83ef2c0b596\" (UID: \"64362e63-6460-4e1c-9827-d83ef2c0b596\") " Oct 29 21:49:57.159494 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.159344 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64362e63-6460-4e1c-9827-d83ef2c0b596-tls-cert\" (UniqueName: \"kubernetes.io/secret/64362e63-6460-4e1c-9827-d83ef2c0b596-tls-cert\") pod \"64362e63-6460-4e1c-9827-d83ef2c0b596\" (UID: \"64362e63-6460-4e1c-9827-d83ef2c0b596\") " Oct 29 21:49:57.161599 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.161570 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64362e63-6460-4e1c-9827-d83ef2c0b596-kube-api-access-wkfxb" pod "64362e63-6460-4e1c-9827-d83ef2c0b596" (UID: "64362e63-6460-4e1c-9827-d83ef2c0b596"). InnerVolumeSpecName "kube-api-access-wkfxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:49:57.161709 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.161589 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64362e63-6460-4e1c-9827-d83ef2c0b596-tls-cert" pod "64362e63-6460-4e1c-9827-d83ef2c0b596" (UID: "64362e63-6460-4e1c-9827-d83ef2c0b596"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:49:57.239183 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.239065 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:49:57.239484 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.239457 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ftwkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Oct 29 21:49:57.239683 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.239536 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" podUID="ebc99afd-f631-46a3-8d94-bfaae865a10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Oct 29 21:49:57.260812 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.260787 4952 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64362e63-6460-4e1c-9827-d83ef2c0b596-tls-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:49:57.260812 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:57.260811 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkfxb\" (UniqueName: \"kubernetes.io/projected/64362e63-6460-4e1c-9827-d83ef2c0b596-kube-api-access-wkfxb\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:49:58.078438 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.078392 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" event={"ID":"ba84b25a-dfd9-4cb8-9316-f29128e39102","Type":"ContainerStarted","Data":"1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734"} Oct 29 21:49:58.078968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.078438 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-zbqld" Oct 29 21:49:58.078968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.078488 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" podUID="ba84b25a-dfd9-4cb8-9316-f29128e39102" containerName="authorino" containerID="cri-o://1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734" gracePeriod=30 Oct 29 21:49:58.142928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.142229 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" podStartSLOduration=2.54717515 podStartE2EDuration="3.142211006s" totalImagesPullingTime="595.035856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:58.140198858 +0000 UTC m=+463.614787280" watchObservedRunningTime="2026-10-29 21:49:58.142211006 +0000 UTC m=+463.616799426" Oct 29 21:49:58.204383 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.204355 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-zbqld"] Oct 29 21:49:58.211878 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.211847 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-zbqld"] Oct 29 21:49:58.247785 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.247289 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ftwkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Oct 29 21:49:58.247785 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.247360 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" podUID="ebc99afd-f631-46a3-8d94-bfaae865a10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Oct 29 21:49:58.395100 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.395078 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" Oct 29 21:49:58.473015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.472969 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba84b25a-dfd9-4cb8-9316-f29128e39102-kube-api-access-pq9tb\" (UniqueName: \"kubernetes.io/projected/ba84b25a-dfd9-4cb8-9316-f29128e39102-kube-api-access-pq9tb\") pod \"ba84b25a-dfd9-4cb8-9316-f29128e39102\" (UID: \"ba84b25a-dfd9-4cb8-9316-f29128e39102\") " Oct 29 21:49:58.475146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.475092 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba84b25a-dfd9-4cb8-9316-f29128e39102-kube-api-access-pq9tb" pod "ba84b25a-dfd9-4cb8-9316-f29128e39102" (UID: "ba84b25a-dfd9-4cb8-9316-f29128e39102"). InnerVolumeSpecName "kube-api-access-pq9tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:49:58.573951 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.573916 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq9tb\" (UniqueName: \"kubernetes.io/projected/ba84b25a-dfd9-4cb8-9316-f29128e39102-kube-api-access-pq9tb\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:49:58.653197 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:58.653117 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64362e63-6460-4e1c-9827-d83ef2c0b596" path="/var/lib/kubelet/pods/64362e63-6460-4e1c-9827-d83ef2c0b596/volumes" Oct 29 21:49:59.085990 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.085957 4952 generic.go:356] "Generic (PLEG): container finished" podID="ba84b25a-dfd9-4cb8-9316-f29128e39102" containerID="1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734" exitCode=0 Oct 29 21:49:59.086398 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.085999 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" Oct 29 21:49:59.086398 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.086051 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" event={"ID":"ba84b25a-dfd9-4cb8-9316-f29128e39102","Type":"ContainerDied","Data":"1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734"} Oct 29 21:49:59.086398 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.086092 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qc9gq" event={"ID":"ba84b25a-dfd9-4cb8-9316-f29128e39102","Type":"ContainerDied","Data":"1bbe495987a7970eba0310ee5d76169585118b710c24ebff91fe8fb8ce83d1b4"} Oct 29 21:49:59.086398 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.086109 4952 scope.go:122] "RemoveContainer" containerID="1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734" Oct 29 21:49:59.094788 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.094763 4952 scope.go:122] "RemoveContainer" containerID="1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734" Oct 29 21:49:59.095094 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:49:59.095072 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734\": container with ID starting with 1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734 not found: ID does not exist" containerID="1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734" Oct 29 21:49:59.095193 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.095100 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734"} err="failed to get container status \"1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734\": rpc error: code = NotFound desc = could not find container \"1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734\": container with ID starting with 1581aa23f801ae159f6703c060d6b7c4c9750b767cfb8caf8f971e4df23b8734 not found: ID does not exist" Oct 29 21:49:59.147898 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.147860 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qc9gq"] Oct 29 21:49:59.149739 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.149716 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qc9gq"] Oct 29 21:49:59.236720 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.236273 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ftwkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Oct 29 21:49:59.236720 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.236339 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" podUID="ebc99afd-f631-46a3-8d94-bfaae865a10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Oct 29 21:49:59.994469 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:49:59.994437 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:50:00.238367 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:00.238329 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ftwkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Oct 29 21:50:00.238990 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:00.238956 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" podUID="ebc99afd-f631-46a3-8d94-bfaae865a10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Oct 29 21:50:00.649823 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:00.649786 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba84b25a-dfd9-4cb8-9316-f29128e39102" path="/var/lib/kubelet/pods/ba84b25a-dfd9-4cb8-9316-f29128e39102/volumes" Oct 29 21:50:01.241202 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:01.241162 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ftwkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Oct 29 21:50:01.241779 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:01.241747 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" podUID="ebc99afd-f631-46a3-8d94-bfaae865a10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Oct 29 21:50:02.240034 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:02.240000 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:50:02.240963 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:02.240940 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ftwkd" Oct 29 21:50:04.927365 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:04.927334 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:50:04.930876 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:04.930855 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hwxzn" Oct 29 21:50:05.856196 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:05.856158 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:50:15.524494 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:50:15.524454 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:50:39.581417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.581328 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg"] Oct 29 21:50:39.581967 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.581765 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ba84b25a-dfd9-4cb8-9316-f29128e39102" containerName="authorino" Oct 29 21:50:39.581967 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.581778 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba84b25a-dfd9-4cb8-9316-f29128e39102" containerName="authorino" Oct 29 21:50:39.581967 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.581862 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba84b25a-dfd9-4cb8-9316-f29128e39102" containerName="authorino" Oct 29 21:50:39.585121 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.585105 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" Oct 29 21:50:39.604592 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.604560 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg"] Oct 29 21:50:39.731953 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.731893 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv45f\" (UniqueName: \"kubernetes.io/projected/ebc631c9-7d1c-450d-ad81-114e2ae2e2b2-kube-api-access-fv45f\") pod \"llmisvc-controller-manager-6466c554d8-x2lkg\" (UID: \"ebc631c9-7d1c-450d-ad81-114e2ae2e2b2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" Oct 29 21:50:39.732161 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.732074 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebc631c9-7d1c-450d-ad81-114e2ae2e2b2-cert\") pod \"llmisvc-controller-manager-6466c554d8-x2lkg\" (UID: \"ebc631c9-7d1c-450d-ad81-114e2ae2e2b2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" Oct 29 21:50:39.833109 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.833017 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv45f\" (UniqueName: \"kubernetes.io/projected/ebc631c9-7d1c-450d-ad81-114e2ae2e2b2-kube-api-access-fv45f\") pod \"llmisvc-controller-manager-6466c554d8-x2lkg\" (UID: \"ebc631c9-7d1c-450d-ad81-114e2ae2e2b2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" Oct 29 21:50:39.833109 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.833089 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebc631c9-7d1c-450d-ad81-114e2ae2e2b2-cert\") pod \"llmisvc-controller-manager-6466c554d8-x2lkg\" (UID: \"ebc631c9-7d1c-450d-ad81-114e2ae2e2b2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" Oct 29 21:50:39.835672 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.835646 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebc631c9-7d1c-450d-ad81-114e2ae2e2b2-cert\") pod \"llmisvc-controller-manager-6466c554d8-x2lkg\" (UID: \"ebc631c9-7d1c-450d-ad81-114e2ae2e2b2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" Oct 29 21:50:39.858381 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.858346 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv45f\" (UniqueName: \"kubernetes.io/projected/ebc631c9-7d1c-450d-ad81-114e2ae2e2b2-kube-api-access-fv45f\") pod \"llmisvc-controller-manager-6466c554d8-x2lkg\" (UID: \"ebc631c9-7d1c-450d-ad81-114e2ae2e2b2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" Oct 29 21:50:39.898202 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:39.898173 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" Oct 29 21:50:40.200945 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:40.200921 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg"] Oct 29 21:50:40.202633 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:50:40.202600 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc631c9_7d1c_450d_ad81_114e2ae2e2b2.slice/crio-7426809e91e6c07e72d7e591018f8d4071245af52227c13c1d5ba3e16fe9c906 WatchSource:0}: Error finding container 7426809e91e6c07e72d7e591018f8d4071245af52227c13c1d5ba3e16fe9c906: Status 404 returned error can't find the container with id 7426809e91e6c07e72d7e591018f8d4071245af52227c13c1d5ba3e16fe9c906 Oct 29 21:50:40.306235 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:40.306199 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" event={"ID":"ebc631c9-7d1c-450d-ad81-114e2ae2e2b2","Type":"ContainerStarted","Data":"7426809e91e6c07e72d7e591018f8d4071245af52227c13c1d5ba3e16fe9c906"} Oct 29 21:50:41.312769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:41.312734 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" event={"ID":"ebc631c9-7d1c-450d-ad81-114e2ae2e2b2","Type":"ContainerStarted","Data":"1daae8aa0c7d5a451979d828887485b71f83a708067599de79757f20e20a0331"} Oct 29 21:50:41.313197 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:41.312799 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" Oct 29 21:50:41.354810 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:50:41.354752 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" podStartSLOduration=2.061583735 podStartE2EDuration="2.354737864s" totalImagesPullingTime="293.154129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:41.354035687 +0000 UTC m=+506.828624130" watchObservedRunningTime="2026-10-29 21:50:41.354737864 +0000 UTC m=+506.829326329" Oct 29 21:50:45.527273 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:50:45.527235 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:51:05.973165 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:05.973129 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:51:12.321131 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:12.321097 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-x2lkg" Oct 29 21:51:12.393013 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:12.392982 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk"] Oct 29 21:51:12.393456 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:12.393411 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" podUID="9e2d99fc-781e-4b37-a85a-a7110cde7c8b" containerName="manager" containerID="cri-o://24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0" gracePeriod=30 Oct 29 21:51:12.633535 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:12.633511 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:51:12.704983 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:12.704953 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-cert\" (UniqueName: \"kubernetes.io/secret/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-cert\") pod \"9e2d99fc-781e-4b37-a85a-a7110cde7c8b\" (UID: \"9e2d99fc-781e-4b37-a85a-a7110cde7c8b\") " Oct 29 21:51:12.705158 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:12.705007 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-kube-api-access-b4rq4\" (UniqueName: \"kubernetes.io/projected/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-kube-api-access-b4rq4\") pod \"9e2d99fc-781e-4b37-a85a-a7110cde7c8b\" (UID: \"9e2d99fc-781e-4b37-a85a-a7110cde7c8b\") " Oct 29 21:51:12.707133 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:12.707103 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-kube-api-access-b4rq4" pod "9e2d99fc-781e-4b37-a85a-a7110cde7c8b" (UID: "9e2d99fc-781e-4b37-a85a-a7110cde7c8b"). InnerVolumeSpecName "kube-api-access-b4rq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:51:12.707234 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:12.707176 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-cert" pod "9e2d99fc-781e-4b37-a85a-a7110cde7c8b" (UID: "9e2d99fc-781e-4b37-a85a-a7110cde7c8b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:51:12.805643 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:12.805613 4952 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:51:12.805643 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:12.805639 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4rq4\" (UniqueName: \"kubernetes.io/projected/9e2d99fc-781e-4b37-a85a-a7110cde7c8b-kube-api-access-b4rq4\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:51:13.477751 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.477713 4952 generic.go:356] "Generic (PLEG): container finished" podID="9e2d99fc-781e-4b37-a85a-a7110cde7c8b" containerID="24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0" exitCode=0 Oct 29 21:51:13.477751 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.477751 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" Oct 29 21:51:13.478309 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.477766 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" event={"ID":"9e2d99fc-781e-4b37-a85a-a7110cde7c8b","Type":"ContainerDied","Data":"24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0"} Oct 29 21:51:13.478309 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.477820 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk" event={"ID":"9e2d99fc-781e-4b37-a85a-a7110cde7c8b","Type":"ContainerDied","Data":"add7481f4a3e2b930527a54b6efb23aef1d3de6f9cf41a83579e433880d3a47b"} Oct 29 21:51:13.478309 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.477850 4952 scope.go:122] "RemoveContainer" containerID="24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0" Oct 29 21:51:13.488425 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.488402 4952 scope.go:122] "RemoveContainer" containerID="24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0" Oct 29 21:51:13.488696 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:51:13.488674 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0\": container with ID starting with 24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0 not found: ID does not exist" containerID="24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0" Oct 29 21:51:13.488757 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.488710 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0"} err="failed to get container status \"24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0\": rpc error: code = NotFound desc = could not find container \"24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0\": container with ID starting with 24b22aef7f40ec4192ddc4ed395f51f2729c43e583bb295a86a40e35f3c095c0 not found: ID does not exist" Oct 29 21:51:13.532705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.532681 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk"] Oct 29 21:51:13.544677 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.544654 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b6vdk"] Oct 29 21:51:13.619128 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.619097 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:51:13.794487 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:13.794390 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:51:14.641620 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:14.641590 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2d99fc-781e-4b37-a85a-a7110cde7c8b" path="/var/lib/kubelet/pods/9e2d99fc-781e-4b37-a85a-a7110cde7c8b/volumes" Oct 29 21:51:15.524675 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:51:15.524637 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:51:18.039267 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:18.039233 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:51:18.225033 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:18.224994 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:51:42.563020 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:42.562925 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:51:42.631188 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:42.631154 4952 generic.go:356] "Generic (PLEG): container finished" podID="bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f" containerID="04fd5ba5f82b3b451af9a0e4ce24a35f893bd8e229e4d23902b75ee6e3b26223" exitCode=1 Oct 29 21:51:42.631348 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:42.631225 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" event={"ID":"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f","Type":"ContainerDied","Data":"04fd5ba5f82b3b451af9a0e4ce24a35f893bd8e229e4d23902b75ee6e3b26223"} Oct 29 21:51:42.631581 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:42.631565 4952 scope.go:122] "RemoveContainer" containerID="04fd5ba5f82b3b451af9a0e4ce24a35f893bd8e229e4d23902b75ee6e3b26223" Oct 29 21:51:43.040962 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:43.040932 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:51:43.638678 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:43.638645 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" event={"ID":"bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f","Type":"ContainerStarted","Data":"51b3e5b64fde5ebe67f9ff6d2a1142d9b7bed28c17cc88dc1dd3ddb208c09fa0"} Oct 29 21:51:43.639099 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:43.638851 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:51:44.388995 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:44.388954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:51:45.525492 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:51:45.525453 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:51:46.630521 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:46.630480 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:51:54.646310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:54.646283 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d8bdd4cf6-g6mpt" Oct 29 21:51:59.104455 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:51:59.104361 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:00.141819 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:00.141783 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:52:02.693385 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:02.693349 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:03.867937 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:03.867879 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:04.782565 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:04.782521 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:52:06.089163 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:06.089126 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:08.142022 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:08.141992 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:10.508832 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:10.508785 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:10.706432 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:10.706395 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:52:13.822149 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:13.822103 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:14.877172 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:14.877130 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 21:52:14.879012 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:14.878973 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 21:52:15.515935 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:15.515882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:15.541353 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:52:15.541286 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:52:15.784823 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:15.784750 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:52:22.524899 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:22.524864 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:52:35.039459 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.039327 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68889bc875-jstq6"] Oct 29 21:52:35.040932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.040394 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9e2d99fc-781e-4b37-a85a-a7110cde7c8b" containerName="manager" Oct 29 21:52:35.040932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.040419 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2d99fc-781e-4b37-a85a-a7110cde7c8b" containerName="manager" Oct 29 21:52:35.040932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.040619 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2d99fc-781e-4b37-a85a-a7110cde7c8b" containerName="manager" Oct 29 21:52:35.044803 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.044777 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68889bc875-jstq6" Oct 29 21:52:35.061434 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.061407 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68889bc875-jstq6"] Oct 29 21:52:35.070990 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.070967 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k7sdt\"" Oct 29 21:52:35.076753 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.076733 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 29 21:52:35.131250 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.131213 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b578808-74eb-41d9-8d5a-cdf40c192710-tls-cert\") pod \"authorino-68889bc875-jstq6\" (UID: \"7b578808-74eb-41d9-8d5a-cdf40c192710\") " pod="kuadrant-system/authorino-68889bc875-jstq6" Oct 29 21:52:35.131431 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.131266 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp88v\" (UniqueName: \"kubernetes.io/projected/7b578808-74eb-41d9-8d5a-cdf40c192710-kube-api-access-zp88v\") pod \"authorino-68889bc875-jstq6\" (UID: \"7b578808-74eb-41d9-8d5a-cdf40c192710\") " pod="kuadrant-system/authorino-68889bc875-jstq6" Oct 29 21:52:35.232201 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.232163 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b578808-74eb-41d9-8d5a-cdf40c192710-tls-cert\") pod \"authorino-68889bc875-jstq6\" (UID: \"7b578808-74eb-41d9-8d5a-cdf40c192710\") " pod="kuadrant-system/authorino-68889bc875-jstq6" Oct 29 21:52:35.232368 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.232223 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp88v\" (UniqueName: \"kubernetes.io/projected/7b578808-74eb-41d9-8d5a-cdf40c192710-kube-api-access-zp88v\") pod \"authorino-68889bc875-jstq6\" (UID: \"7b578808-74eb-41d9-8d5a-cdf40c192710\") " pod="kuadrant-system/authorino-68889bc875-jstq6" Oct 29 21:52:35.234870 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.234846 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b578808-74eb-41d9-8d5a-cdf40c192710-tls-cert\") pod \"authorino-68889bc875-jstq6\" (UID: \"7b578808-74eb-41d9-8d5a-cdf40c192710\") " pod="kuadrant-system/authorino-68889bc875-jstq6" Oct 29 21:52:35.256246 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.256217 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp88v\" (UniqueName: \"kubernetes.io/projected/7b578808-74eb-41d9-8d5a-cdf40c192710-kube-api-access-zp88v\") pod \"authorino-68889bc875-jstq6\" (UID: \"7b578808-74eb-41d9-8d5a-cdf40c192710\") " pod="kuadrant-system/authorino-68889bc875-jstq6" Oct 29 21:52:35.380158 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.380119 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68889bc875-jstq6" Oct 29 21:52:35.685212 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.685183 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68889bc875-jstq6"] Oct 29 21:52:35.940307 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:35.940231 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68889bc875-jstq6" event={"ID":"7b578808-74eb-41d9-8d5a-cdf40c192710","Type":"ContainerStarted","Data":"3b6dd14f6899f31b0d3b44bb1d8b7cd905feef66d11d89e51b1f852d0e29fe18"} Oct 29 21:52:36.099647 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.099611 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:36.284350 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.284324 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:36.929998 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.929967 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x"] Oct 29 21:52:36.933686 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.933662 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:36.948278 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.948207 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:36.948424 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.948299 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwdd4\" (UniqueName: \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-kube-api-access-xwdd4\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:36.948424 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.948365 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d898918-081a-4972-9e4b-2369b33d744e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:36.948523 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.948454 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d898918-081a-4972-9e4b-2369b33d744e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:36.948570 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.948524 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:36.948570 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.948547 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:36.948670 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.948554 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68889bc875-jstq6" event={"ID":"7b578808-74eb-41d9-8d5a-cdf40c192710","Type":"ContainerStarted","Data":"46e7bc977dec27613bba86c6935842afc737ab2e55a2d101d0383bdf51242223"} Oct 29 21:52:36.948670 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.948608 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:36.948780 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.948734 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:36.948828 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.948775 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:36.967641 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.967563 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x"] Oct 29 21:52:36.970883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.970854 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl"] Oct 29 21:52:36.975658 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.975623 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:36.998521 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:36.998494 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl"] Oct 29 21:52:37.030540 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.030480 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68889bc875-jstq6" podStartSLOduration=1.527671387 podStartE2EDuration="2.030460445s" totalImagesPullingTime="502.789058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:37.025270599 +0000 UTC m=+622.499859018" watchObservedRunningTime="2026-10-29 21:52:37.030460445 +0000 UTC m=+622.505048867" Oct 29 21:52:37.049926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.049861 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.050088 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050002 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d898918-081a-4972-9e4b-2369b33d744e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.050088 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050046 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.050195 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050117 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d898918-081a-4972-9e4b-2369b33d744e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.050350 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050334 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.050440 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050367 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.050440 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050402 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.050546 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050445 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.050804 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050781 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.050928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050848 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.050928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050859 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.050928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050880 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.051124 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.050969 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.051124 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.051003 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.051124 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.051062 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.051124 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.051089 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.051124 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.051119 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwdd4\" (UniqueName: \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-kube-api-access-xwdd4\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.051454 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.051145 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7nt\" (UniqueName: \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-kube-api-access-kf7nt\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.051454 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.051177 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.051454 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.051188 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d898918-081a-4972-9e4b-2369b33d744e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.051454 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.051268 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.051760 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.051726 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.051933 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.051894 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.054004 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.053979 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.055708 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.055677 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d898918-081a-4972-9e4b-2369b33d744e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.085954 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.081474 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwdd4\" (UniqueName: \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-kube-api-access-xwdd4\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.085954 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.081963 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-dlq9x\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.152108 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152071 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.152108 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152112 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7nt\" (UniqueName: \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-kube-api-access-kf7nt\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.152634 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152138 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.152634 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152169 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.152634 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152209 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.152634 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152240 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.152634 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152309 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.152634 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152365 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.152634 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152393 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.152634 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152521 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.153080 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152704 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.153080 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.152896 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.153218 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.153194 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.153279 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.153214 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.155121 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.155092 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.155243 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.155138 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.179575 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.179544 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.179575 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.179567 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7nt\" (UniqueName: \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-kube-api-access-kf7nt\") pod \"maas-default-gateway-openshift-default-6f54779779-lvmzl\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.249131 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.249040 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:37.304438 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.304193 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:37.575794 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.575733 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x"] Oct 29 21:52:37.579371 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:52:37.579331 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d898918_081a_4972_9e4b_2369b33d744e.slice/crio-c0c314638616772d17f5c3a8d98770ef654d318b14cb58fdda4d2b8982aca398 WatchSource:0}: Error finding container c0c314638616772d17f5c3a8d98770ef654d318b14cb58fdda4d2b8982aca398: Status 404 returned error can't find the container with id c0c314638616772d17f5c3a8d98770ef654d318b14cb58fdda4d2b8982aca398 Oct 29 21:52:37.582110 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.582073 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:52:37.582210 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.582161 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:52:37.582210 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.582203 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:52:37.664025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.663996 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl"] Oct 29 21:52:37.673957 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.673882 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:52:37.674107 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.674035 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:52:37.674107 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.674080 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:52:37.725295 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.725262 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:37.919257 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.918428 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:52:37.960099 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.960061 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" event={"ID":"78936c9a-79a4-48f5-b4f1-a2087bd3d2da","Type":"ContainerStarted","Data":"cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27"} Oct 29 21:52:37.960700 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.960125 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" event={"ID":"78936c9a-79a4-48f5-b4f1-a2087bd3d2da","Type":"ContainerStarted","Data":"57e871e407126e34cc659091371283883894bbb3eee66dc304ebcd10656c724e"} Oct 29 21:52:37.962456 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.962418 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" event={"ID":"9d898918-081a-4972-9e4b-2369b33d744e","Type":"ContainerStarted","Data":"9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296"} Oct 29 21:52:37.962589 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:37.962476 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" event={"ID":"9d898918-081a-4972-9e4b-2369b33d744e","Type":"ContainerStarted","Data":"c0c314638616772d17f5c3a8d98770ef654d318b14cb58fdda4d2b8982aca398"} Oct 29 21:52:38.012483 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:38.012425 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podStartSLOduration=2.01240588 podStartE2EDuration="2.01240588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:38.008157931 +0000 UTC m=+623.482746372" watchObservedRunningTime="2026-10-29 21:52:38.01240588 +0000 UTC m=+623.486994302" Oct 29 21:52:38.068641 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:38.068585 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" podStartSLOduration=2.06856721 podStartE2EDuration="2.06856721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:38.06656548 +0000 UTC m=+623.541153899" watchObservedRunningTime="2026-10-29 21:52:38.06856721 +0000 UTC m=+623.543155633" Oct 29 21:52:38.257373 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:38.257285 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:38.267604 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:38.267246 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dlq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 29 21:52:38.267604 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:38.267312 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 29 21:52:38.305666 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:38.305602 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:38.307155 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:38.307040 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-lvmzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 29 21:52:38.307155 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:38.307089 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 29 21:52:39.251929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:39.251336 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dlq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 29 21:52:39.251929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:39.251397 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 29 21:52:39.310926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:39.308246 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-lvmzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 29 21:52:39.310926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:39.308310 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 29 21:52:40.252393 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:40.252354 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dlq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 29 21:52:40.252702 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:40.252671 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 29 21:52:40.306646 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:40.306283 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-lvmzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 29 21:52:40.306646 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:40.306335 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 29 21:52:41.254404 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:41.254366 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dlq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 29 21:52:41.255042 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:41.255009 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 29 21:52:41.306114 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:41.306073 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-lvmzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 29 21:52:41.306282 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:41.306157 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 29 21:52:42.253936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:42.253399 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dlq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 29 21:52:42.253936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:42.253480 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 29 21:52:42.306219 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:42.306186 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-lvmzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 29 21:52:42.306631 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:42.306236 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 29 21:52:43.253926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:43.253301 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dlq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 29 21:52:43.253926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:43.253366 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 29 21:52:43.305976 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:43.305929 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-lvmzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 29 21:52:43.306174 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:43.306017 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 29 21:52:44.249740 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:44.249702 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dlq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 29 21:52:44.250201 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:44.249770 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 29 21:52:44.305568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:44.305528 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-lvmzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 29 21:52:44.305768 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:44.305599 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 29 21:52:45.250379 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:45.250341 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-dlq9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 29 21:52:45.250882 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:45.250409 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 29 21:52:45.305563 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:45.305526 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-lvmzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 29 21:52:45.305727 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:45.305588 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 29 21:52:45.530282 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:52:45.530185 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:52:46.443631 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:46.443573 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:46.444411 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:46.444385 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:46.445124 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:46.445103 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 21:52:46.655889 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:46.655818 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:46.656983 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:46.656956 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:52:46.657930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:52:46.657883 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 21:53:04.991776 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:04.991732 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:05.224990 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:05.224943 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:05.407321 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:05.407284 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:05.618499 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:05.618467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:05.829627 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:05.829598 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:06.040360 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:06.039180 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:06.220357 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:06.220272 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:06.403007 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:06.402970 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:06.578689 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:06.578652 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:06.751523 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:06.751490 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:07.190541 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:07.190499 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:07.891869 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:07.891828 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:08.104657 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:08.104612 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:08.857121 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:08.857071 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:09.108291 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:09.108199 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:09.315935 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:09.315886 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:09.499809 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:09.499730 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:10.579146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:10.579109 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:10.752488 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:10.752450 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:11.768735 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:11.768700 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:11.952244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:11.952197 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:12.558699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:12.558666 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:12.822260 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:12.822149 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:13.554750 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:13.554715 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:13.730391 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:13.730353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:14.081395 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:14.081366 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:14.268471 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:14.268433 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:14.611351 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:14.611313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:14.786520 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:14.786481 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:15.020860 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:15.020769 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:15.194927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:15.194868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:15.521379 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:15.521325 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:15.529733 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:53:15.529693 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:53:15.612458 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:15.612427 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:53:15.695671 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:15.695638 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:16.171208 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:16.171172 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:16.378627 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:16.378587 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:17.406696 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.406661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:17.538074 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.538039 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg"] Oct 29 21:53:17.542783 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.542765 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:17.571270 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.571238 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg"] Oct 29 21:53:17.572120 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.572098 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f2459168\"" Oct 29 21:53:17.572375 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.572359 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f2459168-dockercfg-m5hxl\"" Oct 29 21:53:17.585220 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.585174 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:17.615146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.615100 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg\" (UID: \"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:17.615346 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.615176 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gpmx\" (UniqueName: \"kubernetes.io/projected/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-kube-api-access-8gpmx\") pod \"payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg\" (UID: \"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:17.624417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.624383 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb"] Oct 29 21:53:17.628863 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.628821 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:17.662802 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.662729 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb"] Oct 29 21:53:17.716032 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.715989 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg\" (UID: \"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:17.716205 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.716040 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4213a298-d17f-421e-a3b5-45dc2a64c74d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb\" (UID: \"4213a298-d17f-421e-a3b5-45dc2a64c74d\") " pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:17.716205 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.716077 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gpmx\" (UniqueName: \"kubernetes.io/projected/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-kube-api-access-8gpmx\") pod \"payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg\" (UID: \"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:17.716205 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.716137 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476qs\" (UniqueName: \"kubernetes.io/projected/4213a298-d17f-421e-a3b5-45dc2a64c74d-kube-api-access-476qs\") pod \"payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb\" (UID: \"4213a298-d17f-421e-a3b5-45dc2a64c74d\") " pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:17.716697 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.716679 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg\" (UID: \"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:17.740385 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.740357 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gpmx\" (UniqueName: \"kubernetes.io/projected/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-kube-api-access-8gpmx\") pod \"payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg\" (UID: \"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:17.817195 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.817161 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476qs\" (UniqueName: \"kubernetes.io/projected/4213a298-d17f-421e-a3b5-45dc2a64c74d-kube-api-access-476qs\") pod \"payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb\" (UID: \"4213a298-d17f-421e-a3b5-45dc2a64c74d\") " pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:17.817403 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.817228 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4213a298-d17f-421e-a3b5-45dc2a64c74d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb\" (UID: \"4213a298-d17f-421e-a3b5-45dc2a64c74d\") " pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:17.818040 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.818019 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4213a298-d17f-421e-a3b5-45dc2a64c74d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb\" (UID: \"4213a298-d17f-421e-a3b5-45dc2a64c74d\") " pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:17.842760 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.842727 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-476qs\" (UniqueName: \"kubernetes.io/projected/4213a298-d17f-421e-a3b5-45dc2a64c74d-kube-api-access-476qs\") pod \"payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb\" (UID: \"4213a298-d17f-421e-a3b5-45dc2a64c74d\") " pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:17.856423 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.856397 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:17.971788 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:17.969811 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:18.159495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:18.159469 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg"] Oct 29 21:53:18.161213 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:53:18.161184 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6503b8_49e1_4c23_81f4_afe6fc6ce4dd.slice/crio-916f0dab2a2ff13fd4328804f73ecb8b380d87dcffcffffeb9655ef57f5379a9 WatchSource:0}: Error finding container 916f0dab2a2ff13fd4328804f73ecb8b380d87dcffcffffeb9655ef57f5379a9: Status 404 returned error can't find the container with id 916f0dab2a2ff13fd4328804f73ecb8b380d87dcffcffffeb9655ef57f5379a9 Oct 29 21:53:18.167363 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:18.167344 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:53:18.227189 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:18.227156 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" event={"ID":"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd","Type":"ContainerStarted","Data":"916f0dab2a2ff13fd4328804f73ecb8b380d87dcffcffffeb9655ef57f5379a9"} Oct 29 21:53:18.302088 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:53:18.302044 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4213a298_d17f_421e_a3b5_45dc2a64c74d.slice/crio-30be77ffd86e649feaf7f3032e78b2ba09896e9d95957ca6ba4e3874ff3865ce WatchSource:0}: Error finding container 30be77ffd86e649feaf7f3032e78b2ba09896e9d95957ca6ba4e3874ff3865ce: Status 404 returned error can't find the container with id 30be77ffd86e649feaf7f3032e78b2ba09896e9d95957ca6ba4e3874ff3865ce Oct 29 21:53:18.302231 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:18.302209 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb"] Oct 29 21:53:19.243326 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.243281 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" event={"ID":"4213a298-d17f-421e-a3b5-45dc2a64c74d","Type":"ContainerStarted","Data":"30be77ffd86e649feaf7f3032e78b2ba09896e9d95957ca6ba4e3874ff3865ce"} Oct 29 21:53:19.369400 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.369354 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:19.552625 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.552544 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7"] Oct 29 21:53:19.552785 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.552713 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:19.557046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.557016 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 21:53:19.575171 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.575042 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f3b067\"" Oct 29 21:53:19.576132 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.576110 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f3b067-dockercfg-jbx7g\"" Oct 29 21:53:19.580189 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.580165 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7"] Oct 29 21:53:19.760070 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.760029 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bl9s\" (UniqueName: \"kubernetes.io/projected/1ab0a623-d106-4608-817a-6445d1d0cf39-kube-api-access-4bl9s\") pod \"payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7\" (UID: \"1ab0a623-d106-4608-817a-6445d1d0cf39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 21:53:19.760255 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.760165 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ab0a623-d106-4608-817a-6445d1d0cf39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7\" (UID: \"1ab0a623-d106-4608-817a-6445d1d0cf39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 21:53:19.860964 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.860892 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ab0a623-d106-4608-817a-6445d1d0cf39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7\" (UID: \"1ab0a623-d106-4608-817a-6445d1d0cf39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 21:53:19.861217 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.861059 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bl9s\" (UniqueName: \"kubernetes.io/projected/1ab0a623-d106-4608-817a-6445d1d0cf39-kube-api-access-4bl9s\") pod \"payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7\" (UID: \"1ab0a623-d106-4608-817a-6445d1d0cf39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 21:53:19.861693 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.861670 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ab0a623-d106-4608-817a-6445d1d0cf39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7\" (UID: \"1ab0a623-d106-4608-817a-6445d1d0cf39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 21:53:19.886748 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:19.886717 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bl9s\" (UniqueName: \"kubernetes.io/projected/1ab0a623-d106-4608-817a-6445d1d0cf39-kube-api-access-4bl9s\") pod \"payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7\" (UID: \"1ab0a623-d106-4608-817a-6445d1d0cf39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 21:53:20.179490 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:20.179401 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 21:53:20.509356 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:20.509263 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7"] Oct 29 21:53:20.512243 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:53:20.512209 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab0a623_d106_4608_817a_6445d1d0cf39.slice/crio-0e81b932413a9c40274d782e35f9e1b59ba5b8595f3413c194190881d3b79bda WatchSource:0}: Error finding container 0e81b932413a9c40274d782e35f9e1b59ba5b8595f3413c194190881d3b79bda: Status 404 returned error can't find the container with id 0e81b932413a9c40274d782e35f9e1b59ba5b8595f3413c194190881d3b79bda Oct 29 21:53:20.640894 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:20.640859 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:20.898441 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:20.898411 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:21.231195 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.231110 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:21.263935 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.263862 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" event={"ID":"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd","Type":"ContainerStarted","Data":"a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392"} Oct 29 21:53:21.265265 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.263994 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:21.266137 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.266108 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" event={"ID":"1ab0a623-d106-4608-817a-6445d1d0cf39","Type":"ContainerStarted","Data":"95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19"} Oct 29 21:53:21.266327 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.266292 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" event={"ID":"1ab0a623-d106-4608-817a-6445d1d0cf39","Type":"ContainerStarted","Data":"0e81b932413a9c40274d782e35f9e1b59ba5b8595f3413c194190881d3b79bda"} Oct 29 21:53:21.266679 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.266660 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 21:53:21.268623 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.268595 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" event={"ID":"4213a298-d17f-421e-a3b5-45dc2a64c74d","Type":"ContainerStarted","Data":"fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8"} Oct 29 21:53:21.268755 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.268739 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:21.307469 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.307420 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" podStartSLOduration=1.550658589 podStartE2EDuration="4.307404749s" totalImagesPullingTime="2.75674616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:21.305847539 +0000 UTC m=+666.780435957" watchObservedRunningTime="2026-10-29 21:53:21.307404749 +0000 UTC m=+666.781993165" Oct 29 21:53:21.349469 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.349417 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" podStartSLOduration=1.6260933290000001 podStartE2EDuration="4.349402379s" totalImagesPullingTime="2.72330905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:21.348103176 +0000 UTC m=+666.822691593" watchObservedRunningTime="2026-10-29 21:53:21.349402379 +0000 UTC m=+666.823990842" Oct 29 21:53:21.391084 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.391033 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" podStartSLOduration=1.981539945 podStartE2EDuration="2.391019183s" totalImagesPullingTime="409.479238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:21.388464336 +0000 UTC m=+666.863052767" watchObservedRunningTime="2026-10-29 21:53:21.391019183 +0000 UTC m=+666.865607599" Oct 29 21:53:21.402289 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:21.402261 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:22.847529 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.847486 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8"] Oct 29 21:53:22.851893 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.851863 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.879458 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.879425 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:53:22.879644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.879455 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:53:22.879644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.879455 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f2459168-serving-cert\"" Oct 29 21:53:22.879644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.879502 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b8hhr\"" Oct 29 21:53:22.879644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.879609 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 21:53:22.883979 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.883955 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8"] Oct 29 21:53:22.892726 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.892695 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-maas-api-tls\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.892872 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.892753 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhsmj\" (UniqueName: \"kubernetes.io/projected/8fb2f634-2493-415d-b399-3d305840b4a3-kube-api-access-mhsmj\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.892872 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.892833 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb2f634-2493-415d-b399-3d305840b4a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.893012 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.892872 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-metrics-certs\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.993375 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.993343 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-maas-api-tls\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.993535 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.993396 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhsmj\" (UniqueName: \"kubernetes.io/projected/8fb2f634-2493-415d-b399-3d305840b4a3-kube-api-access-mhsmj\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.993535 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.993464 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb2f634-2493-415d-b399-3d305840b4a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.993535 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.993505 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-metrics-certs\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.994145 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.994123 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb2f634-2493-415d-b399-3d305840b4a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.995830 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.995804 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-maas-api-tls\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:22.995942 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:22.995871 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-metrics-certs\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:23.006480 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:23.006453 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:23.018787 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:23.018733 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhsmj\" (UniqueName: \"kubernetes.io/projected/8fb2f634-2493-415d-b399-3d305840b4a3-kube-api-access-mhsmj\") pod \"maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:23.167777 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:23.167693 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:23.194414 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:23.194376 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:23.376176 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:23.376139 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:23.394081 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:23.394051 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:53:23.478500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:23.478417 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8"] Oct 29 21:53:24.293290 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:24.293253 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" event={"ID":"8fb2f634-2493-415d-b399-3d305840b4a3","Type":"ContainerStarted","Data":"9d799ae068fa889a2d818a6b6dc21616ed1dde4ccd34ed700b1e34d19421d872"} Oct 29 21:53:24.941585 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:24.941536 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:25.724326 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:25.724296 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:25.903013 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:25.902795 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:26.316204 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:26.316168 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" event={"ID":"8fb2f634-2493-415d-b399-3d305840b4a3","Type":"ContainerStarted","Data":"223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab"} Oct 29 21:53:26.316377 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:26.316254 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:26.360266 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:26.360215 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" podStartSLOduration=2.537343549 podStartE2EDuration="4.360202193s" totalImagesPullingTime="1.822858644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:26.357733135 +0000 UTC m=+671.832321551" watchObservedRunningTime="2026-10-29 21:53:26.360202193 +0000 UTC m=+671.834790610" Oct 29 21:53:26.562014 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:26.561974 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:28.324631 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:28.324588 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:28.623883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:28.623798 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:53:29.935662 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:29.935628 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x"] Oct 29 21:53:29.939500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:29.939480 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:29.959137 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:29.959114 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x"] Oct 29 21:53:29.967355 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:29.967324 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:29.967573 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:29.967489 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lt7\" (UniqueName: \"kubernetes.io/projected/9c830910-dafd-4b82-a31f-71f207509e55-kube-api-access-p9lt7\") pod \"maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:29.967672 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:29.967636 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:29.967936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:29.967915 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:53:29.968062 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:29.968011 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:53:30.068735 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:30.068707 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lt7\" (UniqueName: \"kubernetes.io/projected/9c830910-dafd-4b82-a31f-71f207509e55-kube-api-access-p9lt7\") pod \"maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:30.068912 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:30.068776 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:30.068912 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:30.068811 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:30.069436 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:30.069408 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:30.071149 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:30.071126 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:30.093535 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:30.093504 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lt7\" (UniqueName: \"kubernetes.io/projected/9c830910-dafd-4b82-a31f-71f207509e55-kube-api-access-p9lt7\") pod \"maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:30.124668 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:30.124630 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:30.252974 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:30.252872 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:30.553496 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:30.553467 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x"] Oct 29 21:53:30.554935 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:53:30.554898 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c830910_dafd_4b82_a31f_71f207509e55.slice/crio-eb394bebfc3c6f91aa074ac40c4201e4efb83b902eddccd9a64644305961540d WatchSource:0}: Error finding container eb394bebfc3c6f91aa074ac40c4201e4efb83b902eddccd9a64644305961540d: Status 404 returned error can't find the container with id eb394bebfc3c6f91aa074ac40c4201e4efb83b902eddccd9a64644305961540d Oct 29 21:53:31.351371 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:31.351337 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" event={"ID":"9c830910-dafd-4b82-a31f-71f207509e55","Type":"ContainerStarted","Data":"eb394bebfc3c6f91aa074ac40c4201e4efb83b902eddccd9a64644305961540d"} Oct 29 21:53:31.975421 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:31.975362 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:32.236525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:32.236438 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:32.277587 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:32.277556 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:32.277753 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:32.277643 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:32.349955 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:32.329657 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:53:33.284979 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:33.284871 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 21:53:33.368168 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:33.368135 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" event={"ID":"9c830910-dafd-4b82-a31f-71f207509e55","Type":"ContainerStarted","Data":"62451bae29d351d0689fd4f7e61bf4e63e31ef59171a35069d88111084bc0e40"} Oct 29 21:53:33.413013 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:33.412953 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" podStartSLOduration=2.352563358 podStartE2EDuration="4.412935655s" totalImagesPullingTime="2.060372297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:33.411105018 +0000 UTC m=+678.885693436" watchObservedRunningTime="2026-10-29 21:53:33.412935655 +0000 UTC m=+678.887524070" Oct 29 21:53:33.858600 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:33.858551 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:34.375844 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:34.375814 4952 generic.go:356] "Generic (PLEG): container finished" podID="9c830910-dafd-4b82-a31f-71f207509e55" containerID="62451bae29d351d0689fd4f7e61bf4e63e31ef59171a35069d88111084bc0e40" exitCode=0 Oct 29 21:53:34.376276 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:34.375916 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" event={"ID":"9c830910-dafd-4b82-a31f-71f207509e55","Type":"ContainerDied","Data":"62451bae29d351d0689fd4f7e61bf4e63e31ef59171a35069d88111084bc0e40"} Oct 29 21:53:35.528835 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.528710 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:35.543748 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.543725 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:35.624018 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.623991 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-xks-ca\") pod \"9c830910-dafd-4b82-a31f-71f207509e55\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " Oct 29 21:53:35.624203 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.624096 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-openshift-ca\") pod \"9c830910-dafd-4b82-a31f-71f207509e55\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " Oct 29 21:53:35.624203 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.624120 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c830910-dafd-4b82-a31f-71f207509e55-kube-api-access-p9lt7\" (UniqueName: \"kubernetes.io/projected/9c830910-dafd-4b82-a31f-71f207509e55-kube-api-access-p9lt7\") pod \"9c830910-dafd-4b82-a31f-71f207509e55\" (UID: \"9c830910-dafd-4b82-a31f-71f207509e55\") " Oct 29 21:53:35.624526 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.624497 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-openshift-ca" pod "9c830910-dafd-4b82-a31f-71f207509e55" (UID: "9c830910-dafd-4b82-a31f-71f207509e55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:35.626315 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.626288 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-xks-ca" pod "9c830910-dafd-4b82-a31f-71f207509e55" (UID: "9c830910-dafd-4b82-a31f-71f207509e55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:35.626589 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.626571 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c830910-dafd-4b82-a31f-71f207509e55-kube-api-access-p9lt7" pod "9c830910-dafd-4b82-a31f-71f207509e55" (UID: "9c830910-dafd-4b82-a31f-71f207509e55"). InnerVolumeSpecName "kube-api-access-p9lt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:35.725456 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.725373 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-openshift-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:53:35.725456 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.725410 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lt7\" (UniqueName: \"kubernetes.io/projected/9c830910-dafd-4b82-a31f-71f207509e55-kube-api-access-p9lt7\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:53:35.725456 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:35.725425 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c830910-dafd-4b82-a31f-71f207509e55-maas-api-xks-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:53:36.393487 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:36.393451 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" event={"ID":"9c830910-dafd-4b82-a31f-71f207509e55","Type":"ContainerDied","Data":"eb394bebfc3c6f91aa074ac40c4201e4efb83b902eddccd9a64644305961540d"} Oct 29 21:53:36.393653 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:36.393535 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb394bebfc3c6f91aa074ac40c4201e4efb83b902eddccd9a64644305961540d" Oct 29 21:53:36.393653 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:36.393460 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x" Oct 29 21:53:36.438072 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:36.437283 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:53:36.936095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:36.936043 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:37.425172 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:37.425137 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:37.611926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:37.611416 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:37.789988 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:37.789885 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:39.150010 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.149972 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:39.254367 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.254327 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk"] Oct 29 21:53:39.255101 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.255077 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9c830910-dafd-4b82-a31f-71f207509e55" containerName="revoke-keys" Oct 29 21:53:39.255278 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.255254 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c830910-dafd-4b82-a31f-71f207509e55" containerName="revoke-keys" Oct 29 21:53:39.255491 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.255475 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c830910-dafd-4b82-a31f-71f207509e55" containerName="revoke-keys" Oct 29 21:53:39.261226 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.261204 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.279480 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.279456 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk"] Oct 29 21:53:39.288072 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.288041 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f3b067\"/\"kube-root-ca.crt\"" Oct 29 21:53:39.288177 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.288067 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f3b067\"/\"openshift-service-ca.crt\"" Oct 29 21:53:39.288883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.288866 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3b067\"/\"simulator-w0-f3b067-kserve-self-signed-certs\"" Oct 29 21:53:39.288964 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.288871 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3b067\"/\"default-dockercfg-xqpdv\"" Oct 29 21:53:39.393507 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.393469 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d67wv\" (UniqueName: \"kubernetes.io/projected/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-kube-api-access-d67wv\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.393687 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.393523 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-home\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.393687 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.393641 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-model-cache\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.393806 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.393722 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tmp-dir\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.393856 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.393807 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-dshm\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.393952 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.393930 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tls-certs\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.467444 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.467342 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:39.498462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.495568 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-dshm\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.498462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.495661 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tls-certs\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.498462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.495736 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d67wv\" (UniqueName: \"kubernetes.io/projected/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-kube-api-access-d67wv\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.498462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.495770 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-home\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.498462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.495818 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-model-cache\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.498462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.495848 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tmp-dir\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.498462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.496270 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tmp-dir\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.498462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.496527 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-home\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.498462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.496757 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-model-cache\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.499130 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.498788 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-dshm\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.507308 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.500761 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tls-certs\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.520968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.520936 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d67wv\" (UniqueName: \"kubernetes.io/projected/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-kube-api-access-d67wv\") pod \"simulator-w0-f3b067-kserve-6d859fb8fc-t65pk\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.578354 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.578321 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:39.891496 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:39.891469 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk"] Oct 29 21:53:39.892504 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:53:39.892475 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92a3f8a0_1b4d_41d1_8d9b_f0e3aac6c134.slice/crio-ff286d422c2df76a98ca65fcbe8c4a3b7b5ae0196500b56c0c9f341c82119996 WatchSource:0}: Error finding container ff286d422c2df76a98ca65fcbe8c4a3b7b5ae0196500b56c0c9f341c82119996: Status 404 returned error can't find the container with id ff286d422c2df76a98ca65fcbe8c4a3b7b5ae0196500b56c0c9f341c82119996 Oct 29 21:53:40.235270 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.221961 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:40.401032 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.401000 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:40.421774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.421727 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" event={"ID":"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134","Type":"ContainerStarted","Data":"ff286d422c2df76a98ca65fcbe8c4a3b7b5ae0196500b56c0c9f341c82119996"} Oct 29 21:53:40.563922 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.563827 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5"] Oct 29 21:53:40.570806 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.570784 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.586641 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.586603 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5"] Oct 29 21:53:40.596432 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.595107 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 21:53:40.596432 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.595480 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 21:53:40.709324 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.709281 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85g72\" (UniqueName: \"kubernetes.io/projected/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-kube-api-access-85g72\") pod \"maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.709508 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.709381 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.709508 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.709417 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.727041 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.727009 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:40.813458 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.810553 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.813458 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.810606 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.813458 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.810724 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85g72\" (UniqueName: \"kubernetes.io/projected/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-kube-api-access-85g72\") pod \"maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.813458 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.811930 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.820926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.814874 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.841387 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.841359 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85g72\" (UniqueName: \"kubernetes.io/projected/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-kube-api-access-85g72\") pod \"maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.888144 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.888113 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:40.926799 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:40.926759 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:41.122527 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:41.122464 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:41.216558 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:41.216527 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5"] Oct 29 21:53:41.218250 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:53:41.218218 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6311d3f6_92a6_48d8_ab2a_5e8c00382b25.slice/crio-86a70cab116bbde9ec32c5cda09478708bd0ea30e454bfd84529c154b4385e76 WatchSource:0}: Error finding container 86a70cab116bbde9ec32c5cda09478708bd0ea30e454bfd84529c154b4385e76: Status 404 returned error can't find the container with id 86a70cab116bbde9ec32c5cda09478708bd0ea30e454bfd84529c154b4385e76 Oct 29 21:53:41.430984 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:41.430952 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" event={"ID":"6311d3f6-92a6-48d8-ab2a-5e8c00382b25","Type":"ContainerStarted","Data":"86a70cab116bbde9ec32c5cda09478708bd0ea30e454bfd84529c154b4385e76"} Oct 29 21:53:41.546876 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:41.546792 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:42.438756 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:42.438722 4952 generic.go:356] "Generic (PLEG): container finished" podID="6311d3f6-92a6-48d8-ab2a-5e8c00382b25" containerID="815de20d306c2030d94da07c355b9854d9508c18c8978e6b22bd6e5c73ec0dd5" exitCode=0 Oct 29 21:53:42.439231 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:42.438811 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" event={"ID":"6311d3f6-92a6-48d8-ab2a-5e8c00382b25","Type":"ContainerDied","Data":"815de20d306c2030d94da07c355b9854d9508c18c8978e6b22bd6e5c73ec0dd5"} Oct 29 21:53:42.440635 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:42.440603 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" event={"ID":"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134","Type":"ContainerStarted","Data":"6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a"} Oct 29 21:53:42.525997 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:42.525892 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podStartSLOduration=2.072667086 podStartE2EDuration="3.525877663s" totalImagesPullingTime="1.453210577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:42.523408557 +0000 UTC m=+687.997996974" watchObservedRunningTime="2026-10-29 21:53:42.525877663 +0000 UTC m=+688.000466080" Oct 29 21:53:42.579419 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:42.579384 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:43.234927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.233665 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:53:43.325065 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.325028 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:43.629683 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.629655 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:43.748932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.748889 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-kube-api-access-85g72\" (UniqueName: \"kubernetes.io/projected/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-kube-api-access-85g72\") pod \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " Oct 29 21:53:43.749097 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.748966 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-openshift-ca\") pod \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " Oct 29 21:53:43.749097 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.749064 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-xks-ca\") pod \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\" (UID: \"6311d3f6-92a6-48d8-ab2a-5e8c00382b25\") " Oct 29 21:53:43.749397 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.749361 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-openshift-ca" pod "6311d3f6-92a6-48d8-ab2a-5e8c00382b25" (UID: "6311d3f6-92a6-48d8-ab2a-5e8c00382b25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:43.751185 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.751161 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-xks-ca" pod "6311d3f6-92a6-48d8-ab2a-5e8c00382b25" (UID: "6311d3f6-92a6-48d8-ab2a-5e8c00382b25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:53:43.751282 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.751264 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-kube-api-access-85g72" pod "6311d3f6-92a6-48d8-ab2a-5e8c00382b25" (UID: "6311d3f6-92a6-48d8-ab2a-5e8c00382b25"). InnerVolumeSpecName "kube-api-access-85g72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:43.850722 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.850681 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-xks-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:53:43.850722 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.850711 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85g72\" (UniqueName: \"kubernetes.io/projected/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-kube-api-access-85g72\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:53:43.850722 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.850722 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6311d3f6-92a6-48d8-ab2a-5e8c00382b25-maas-api-openshift-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:53:43.914755 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.914723 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:43.977273 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.977242 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x"] Oct 29 21:53:43.981516 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:43.981479 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229ab2ac-xhx9x"] Oct 29 21:53:44.093108 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:44.093077 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:44.455343 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:44.455309 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" Oct 29 21:53:44.455343 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:44.455324 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5" event={"ID":"6311d3f6-92a6-48d8-ab2a-5e8c00382b25","Type":"ContainerDied","Data":"86a70cab116bbde9ec32c5cda09478708bd0ea30e454bfd84529c154b4385e76"} Oct 29 21:53:44.455603 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:44.455369 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86a70cab116bbde9ec32c5cda09478708bd0ea30e454bfd84529c154b4385e76" Oct 29 21:53:44.655780 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:44.655372 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c830910-dafd-4b82-a31f-71f207509e55" path="/var/lib/kubelet/pods/9c830910-dafd-4b82-a31f-71f207509e55/volumes" Oct 29 21:53:44.755624 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:44.755526 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:45.223049 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:45.223011 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:45.530294 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:53:45.530211 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:53:45.603101 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:45.603064 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:46.163669 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:46.163618 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:46.595425 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:46.595375 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:46.963072 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:46.962994 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:48.635794 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:48.635746 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:49.335203 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:49.335137 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:49.507102 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:49.507061 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:49.578669 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:49.578635 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:49.592429 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:49.592358 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:50.064804 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:50.064690 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:50.250565 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:50.250515 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:50.447058 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:50.447016 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:50.500918 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:50.500878 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:53:50.617318 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:50.617272 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:51.003929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:51.003864 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:51.184801 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:51.184767 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:51.372432 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:51.372391 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:51.569194 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:51.569154 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:51.942391 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:51.942359 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:52.126712 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:52.126681 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:52.519360 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:52.519321 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:52.703597 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:52.703557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:52.736376 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:52.736334 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb"] Oct 29 21:53:52.774241 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:52.774163 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg"] Oct 29 21:53:52.913754 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:52.913714 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:53.093979 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.093946 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:53.403583 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.403494 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv"] Oct 29 21:53:53.404171 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.404134 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6311d3f6-92a6-48d8-ab2a-5e8c00382b25" containerName="revoke-keys" Oct 29 21:53:53.404171 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.404163 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6311d3f6-92a6-48d8-ab2a-5e8c00382b25" containerName="revoke-keys" Oct 29 21:53:53.404394 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.404276 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6311d3f6-92a6-48d8-ab2a-5e8c00382b25" containerName="revoke-keys" Oct 29 21:53:53.410409 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.410380 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.426993 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.426964 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"openshift-service-ca.crt\"" Oct 29 21:53:53.427501 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.427483 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"kube-root-ca.crt\"" Oct 29 21:53:53.429590 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.429559 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv"] Oct 29 21:53:53.439184 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.439161 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"default-dockercfg-58jq5\"" Oct 29 21:53:53.439352 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.439285 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"simulator-w1-29c2f2-kserve-self-signed-certs\"" Oct 29 21:53:53.453680 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.453643 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-home\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.453853 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.453707 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl4jf\" (UniqueName: \"kubernetes.io/projected/de8d8749-f5c9-4bbc-870a-bd919a44dec7-kube-api-access-gl4jf\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.453853 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.453780 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tls-certs\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.454026 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.453865 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tmp-dir\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.454026 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.453941 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-model-cache\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.454026 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.453979 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-dshm\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.507113 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.507080 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:53.554769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.554732 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-dshm\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.555323 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.554807 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-home\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.555323 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.554876 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4jf\" (UniqueName: \"kubernetes.io/projected/de8d8749-f5c9-4bbc-870a-bd919a44dec7-kube-api-access-gl4jf\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.555323 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.554930 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tls-certs\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.555323 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.554968 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tmp-dir\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.555323 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.555002 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-model-cache\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.555628 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.555373 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-model-cache\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.555828 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.555800 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tmp-dir\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.555894 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.555863 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-home\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.557585 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.557555 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-dshm\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.557749 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.557628 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tls-certs\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.579457 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.579430 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl4jf\" (UniqueName: \"kubernetes.io/projected/de8d8749-f5c9-4bbc-870a-bd919a44dec7-kube-api-access-gl4jf\") pod \"simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:53.681777 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.681671 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:53.724950 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:53.724913 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:54.039819 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:54.039784 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv"] Oct 29 21:53:54.040209 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:53:54.040159 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde8d8749_f5c9_4bbc_870a_bd919a44dec7.slice/crio-f75f3e71f54ed07733509cfb0250ece1197d4212034b6a219f14eff1e6d99d45 WatchSource:0}: Error finding container f75f3e71f54ed07733509cfb0250ece1197d4212034b6a219f14eff1e6d99d45: Status 404 returned error can't find the container with id f75f3e71f54ed07733509cfb0250ece1197d4212034b6a219f14eff1e6d99d45 Oct 29 21:53:54.104945 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:54.101823 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:54.291026 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:54.290939 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:54.517772 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:54.517737 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" event={"ID":"de8d8749-f5c9-4bbc-870a-bd919a44dec7","Type":"ContainerStarted","Data":"28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490"} Oct 29 21:53:54.517994 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:54.517793 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" event={"ID":"de8d8749-f5c9-4bbc-870a-bd919a44dec7","Type":"ContainerStarted","Data":"f75f3e71f54ed07733509cfb0250ece1197d4212034b6a219f14eff1e6d99d45"} Oct 29 21:53:54.560920 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:54.560784 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podStartSLOduration=1.560765067 podStartE2EDuration="1.560765067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:54.559853601 +0000 UTC m=+700.034442020" watchObservedRunningTime="2026-10-29 21:53:54.560765067 +0000 UTC m=+700.035353486" Oct 29 21:53:54.620324 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:54.620290 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:54.725488 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:54.725454 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:53:56.426193 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:56.426152 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:53:56.593982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:56.593946 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:57.401246 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:57.401204 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:57.574614 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:57.574569 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:57.737298 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:57.737207 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" podUID="4213a298-d17f-421e-a3b5-45dc2a64c74d" containerName="payload-processing" containerID="cri-o://fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8" gracePeriod=25 Oct 29 21:53:57.774821 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:57.774776 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" podUID="9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd" containerName="payload-pre-processing" containerID="cri-o://a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392" gracePeriod=25 Oct 29 21:53:58.041338 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.041249 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:58.111396 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.111370 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:58.118465 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.118439 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4213a298-d17f-421e-a3b5-45dc2a64c74d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4213a298-d17f-421e-a3b5-45dc2a64c74d-plugins-config-volume\") pod \"4213a298-d17f-421e-a3b5-45dc2a64c74d\" (UID: \"4213a298-d17f-421e-a3b5-45dc2a64c74d\") " Oct 29 21:53:58.118609 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.118543 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4213a298-d17f-421e-a3b5-45dc2a64c74d-kube-api-access-476qs\" (UniqueName: \"kubernetes.io/projected/4213a298-d17f-421e-a3b5-45dc2a64c74d-kube-api-access-476qs\") pod \"4213a298-d17f-421e-a3b5-45dc2a64c74d\" (UID: \"4213a298-d17f-421e-a3b5-45dc2a64c74d\") " Oct 29 21:53:58.118841 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.118816 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4213a298-d17f-421e-a3b5-45dc2a64c74d-plugins-config-volume" pod "4213a298-d17f-421e-a3b5-45dc2a64c74d" (UID: "4213a298-d17f-421e-a3b5-45dc2a64c74d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:58.120737 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.120710 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4213a298-d17f-421e-a3b5-45dc2a64c74d-kube-api-access-476qs" pod "4213a298-d17f-421e-a3b5-45dc2a64c74d" (UID: "4213a298-d17f-421e-a3b5-45dc2a64c74d"). InnerVolumeSpecName "kube-api-access-476qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:58.189943 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.189880 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:58.220055 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.220026 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-plugins-config-volume\") pod \"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd\" (UID: \"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd\") " Oct 29 21:53:58.220223 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.220071 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-kube-api-access-8gpmx\" (UniqueName: \"kubernetes.io/projected/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-kube-api-access-8gpmx\") pod \"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd\" (UID: \"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd\") " Oct 29 21:53:58.220423 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.220404 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-476qs\" (UniqueName: \"kubernetes.io/projected/4213a298-d17f-421e-a3b5-45dc2a64c74d-kube-api-access-476qs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.220473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.220430 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4213a298-d17f-421e-a3b5-45dc2a64c74d-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.220508 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.220477 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-plugins-config-volume" pod "9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd" (UID: "9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:53:58.222305 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.222277 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-kube-api-access-8gpmx" pod "9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd" (UID: "9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd"). InnerVolumeSpecName "kube-api-access-8gpmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:53:58.321367 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.321268 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.321367 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.321301 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gpmx\" (UniqueName: \"kubernetes.io/projected/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd-kube-api-access-8gpmx\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:53:58.551095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.551060 4952 generic.go:356] "Generic (PLEG): container finished" podID="9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd" containerID="a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392" exitCode=0 Oct 29 21:53:58.551283 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.551130 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" Oct 29 21:53:58.551283 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.551155 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" event={"ID":"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd","Type":"ContainerDied","Data":"a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392"} Oct 29 21:53:58.551283 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.551262 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg" event={"ID":"9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd","Type":"ContainerDied","Data":"916f0dab2a2ff13fd4328804f73ecb8b380d87dcffcffffeb9655ef57f5379a9"} Oct 29 21:53:58.551507 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.551297 4952 scope.go:122] "RemoveContainer" containerID="a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392" Oct 29 21:53:58.554276 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.553616 4952 generic.go:356] "Generic (PLEG): container finished" podID="4213a298-d17f-421e-a3b5-45dc2a64c74d" containerID="fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8" exitCode=0 Oct 29 21:53:58.554276 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.553653 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" event={"ID":"4213a298-d17f-421e-a3b5-45dc2a64c74d","Type":"ContainerDied","Data":"fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8"} Oct 29 21:53:58.554276 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.553686 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" Oct 29 21:53:58.554276 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.553695 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb" event={"ID":"4213a298-d17f-421e-a3b5-45dc2a64c74d","Type":"ContainerDied","Data":"30be77ffd86e649feaf7f3032e78b2ba09896e9d95957ca6ba4e3874ff3865ce"} Oct 29 21:53:58.563492 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.563340 4952 scope.go:122] "RemoveContainer" containerID="a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392" Oct 29 21:53:58.564002 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:53:58.563974 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392\": container with ID starting with a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392 not found: ID does not exist" containerID="a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392" Oct 29 21:53:58.564102 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.564015 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392"} err="failed to get container status \"a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392\": rpc error: code = NotFound desc = could not find container \"a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392\": container with ID starting with a77baee515a51d939e8b670029d358e94bf41ce44eb5d1913cc7985d18b9a392 not found: ID does not exist" Oct 29 21:53:58.564102 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.564044 4952 scope.go:122] "RemoveContainer" containerID="fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8" Oct 29 21:53:58.575465 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.575446 4952 scope.go:122] "RemoveContainer" containerID="fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8" Oct 29 21:53:58.575809 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:53:58.575790 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8\": container with ID starting with fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8 not found: ID does not exist" containerID="fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8" Oct 29 21:53:58.575855 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.575818 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8"} err="failed to get container status \"fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8\": rpc error: code = NotFound desc = could not find container \"fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8\": container with ID starting with fce787d57e703b133efd95c06fa9fe8c2160f19c29b6a9f70c941232bd8f13e8 not found: ID does not exist" Oct 29 21:53:58.610784 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.610744 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg"] Oct 29 21:53:58.623487 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.623457 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f2459168-7ff8cb8775-cbrhg"] Oct 29 21:53:58.651146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.651112 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd" path="/var/lib/kubelet/pods/9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd/volumes" Oct 29 21:53:58.683405 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.683369 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb"] Oct 29 21:53:58.691353 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.691317 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f2459168-6794fc769d-xbcjb"] Oct 29 21:53:58.775435 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.775393 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:58.946368 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:58.946292 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:59.173263 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:59.173227 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:59.368407 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:59.368310 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:59.539138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:59.539096 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:53:59.949203 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:53:59.949166 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:00.135007 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.134971 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:00.318746 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.318655 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:00.495994 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.495953 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:00.565765 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.565576 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8"] Oct 29 21:54:00.565970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.565818 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" podUID="8fb2f634-2493-415d-b399-3d305840b4a3" containerName="maas-api" containerID="cri-o://223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab" gracePeriod=30 Oct 29 21:54:00.645653 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.645617 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4213a298-d17f-421e-a3b5-45dc2a64c74d" path="/var/lib/kubelet/pods/4213a298-d17f-421e-a3b5-45dc2a64c74d/volumes" Oct 29 21:54:00.684545 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.684508 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:00.861303 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.861269 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:00.870554 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.870528 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:54:00.955937 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.943752 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-metrics-certs\") pod \"8fb2f634-2493-415d-b399-3d305840b4a3\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " Oct 29 21:54:00.955937 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.943805 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-maas-api-tls\") pod \"8fb2f634-2493-415d-b399-3d305840b4a3\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " Oct 29 21:54:00.955937 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.943965 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb2f634-2493-415d-b399-3d305840b4a3-kube-api-access-mhsmj\" (UniqueName: \"kubernetes.io/projected/8fb2f634-2493-415d-b399-3d305840b4a3-kube-api-access-mhsmj\") pod \"8fb2f634-2493-415d-b399-3d305840b4a3\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " Oct 29 21:54:00.955937 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.944007 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb2f634-2493-415d-b399-3d305840b4a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb2f634-2493-415d-b399-3d305840b4a3-openshift-service-ca\") pod \"8fb2f634-2493-415d-b399-3d305840b4a3\" (UID: \"8fb2f634-2493-415d-b399-3d305840b4a3\") " Oct 29 21:54:00.955937 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.944598 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb2f634-2493-415d-b399-3d305840b4a3-openshift-service-ca" pod "8fb2f634-2493-415d-b399-3d305840b4a3" (UID: "8fb2f634-2493-415d-b399-3d305840b4a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 21:54:00.973936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.959045 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-maas-api-tls" pod "8fb2f634-2493-415d-b399-3d305840b4a3" (UID: "8fb2f634-2493-415d-b399-3d305840b4a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:54:00.973936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.963016 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb2f634-2493-415d-b399-3d305840b4a3-kube-api-access-mhsmj" pod "8fb2f634-2493-415d-b399-3d305840b4a3" (UID: "8fb2f634-2493-415d-b399-3d305840b4a3"). InnerVolumeSpecName "kube-api-access-mhsmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:54:00.980972 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:00.975041 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-metrics-certs" pod "8fb2f634-2493-415d-b399-3d305840b4a3" (UID: "8fb2f634-2493-415d-b399-3d305840b4a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:54:01.044973 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.044933 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhsmj\" (UniqueName: \"kubernetes.io/projected/8fb2f634-2493-415d-b399-3d305840b4a3-kube-api-access-mhsmj\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:54:01.044973 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.044971 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb2f634-2493-415d-b399-3d305840b4a3-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:54:01.045235 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.044986 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:54:01.045235 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.045000 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb2f634-2493-415d-b399-3d305840b4a3-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:54:01.085992 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.085952 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:01.277275 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.277186 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:01.367471 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.366817 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5"] Oct 29 21:54:01.378885 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.375343 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f2459168-lxcq5"] Oct 29 21:54:01.454892 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.454853 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:01.584456 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.584419 4952 generic.go:356] "Generic (PLEG): container finished" podID="8fb2f634-2493-415d-b399-3d305840b4a3" containerID="223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab" exitCode=0 Oct 29 21:54:01.584649 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.584506 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" Oct 29 21:54:01.584649 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.584509 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" event={"ID":"8fb2f634-2493-415d-b399-3d305840b4a3","Type":"ContainerDied","Data":"223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab"} Oct 29 21:54:01.584774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.584655 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8" event={"ID":"8fb2f634-2493-415d-b399-3d305840b4a3","Type":"ContainerDied","Data":"9d799ae068fa889a2d818a6b6dc21616ed1dde4ccd34ed700b1e34d19421d872"} Oct 29 21:54:01.584774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.584694 4952 scope.go:122] "RemoveContainer" containerID="223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab" Oct 29 21:54:01.595771 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.595742 4952 scope.go:122] "RemoveContainer" containerID="223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab" Oct 29 21:54:01.596089 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:54:01.596059 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab\": container with ID starting with 223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab not found: ID does not exist" containerID="223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab" Oct 29 21:54:01.596154 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.596103 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab"} err="failed to get container status \"223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab\": rpc error: code = NotFound desc = could not find container \"223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab\": container with ID starting with 223300007ed27f7504f7b38d4743b14ec514646df948c3ecc45c1a72064d3eab not found: ID does not exist" Oct 29 21:54:01.640006 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.639972 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8"] Oct 29 21:54:01.642272 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.642235 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f2459168-ff8bf9655-nqjm8"] Oct 29 21:54:01.644847 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:01.644819 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:02.564212 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:02.564120 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:02.643445 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:02.643406 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6311d3f6-92a6-48d8-ab2a-5e8c00382b25" path="/var/lib/kubelet/pods/6311d3f6-92a6-48d8-ab2a-5e8c00382b25/volumes" Oct 29 21:54:02.643952 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:02.643928 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb2f634-2493-415d-b399-3d305840b4a3" path="/var/lib/kubelet/pods/8fb2f634-2493-415d-b399-3d305840b4a3/volumes" Oct 29 21:54:02.737228 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:02.737193 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:03.089924 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.082381 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:03.261870 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.261820 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:03.452116 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.452037 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:03.479767 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.479730 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z"] Oct 29 21:54:03.480417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.480396 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd" containerName="payload-pre-processing" Oct 29 21:54:03.480484 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.480421 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd" containerName="payload-pre-processing" Oct 29 21:54:03.480484 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.480481 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8fb2f634-2493-415d-b399-3d305840b4a3" containerName="maas-api" Oct 29 21:54:03.480571 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.480491 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb2f634-2493-415d-b399-3d305840b4a3" containerName="maas-api" Oct 29 21:54:03.480571 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.480507 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4213a298-d17f-421e-a3b5-45dc2a64c74d" containerName="payload-processing" Oct 29 21:54:03.480571 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.480517 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4213a298-d17f-421e-a3b5-45dc2a64c74d" containerName="payload-processing" Oct 29 21:54:03.480681 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.480639 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb2f634-2493-415d-b399-3d305840b4a3" containerName="maas-api" Oct 29 21:54:03.480681 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.480650 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4213a298-d17f-421e-a3b5-45dc2a64c74d" containerName="payload-processing" Oct 29 21:54:03.480681 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.480661 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6503b8-49e1-4c23-81f4-afe6fc6ce4dd" containerName="payload-pre-processing" Oct 29 21:54:03.485802 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.485783 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.506530 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.506501 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z"] Oct 29 21:54:03.514351 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.514316 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3b067\"/\"premium-w0-f3b067-kserve-self-signed-certs\"" Oct 29 21:54:03.573183 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.573143 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-home\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.573183 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.573190 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-model-cache\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.573640 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.573251 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhvx\" (UniqueName: \"kubernetes.io/projected/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-kube-api-access-9mhvx\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.573640 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.573281 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-dshm\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.573640 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.573352 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tls-certs\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.573640 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.573379 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tmp-dir\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.626965 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.624812 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:03.675483 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.675442 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-home\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.675653 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.675495 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-model-cache\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.675653 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.675577 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhvx\" (UniqueName: \"kubernetes.io/projected/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-kube-api-access-9mhvx\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.675653 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.675608 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-dshm\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.675783 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.675694 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tls-certs\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.675783 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.675718 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tmp-dir\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.676217 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.676193 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tmp-dir\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.676516 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.676490 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-home\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.676818 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.676799 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-model-cache\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.682451 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.682422 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tls-certs\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.691392 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.691328 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-dshm\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.702081 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.702043 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhvx\" (UniqueName: \"kubernetes.io/projected/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-kube-api-access-9mhvx\") pod \"premium-w0-f3b067-kserve-7987b66d96-k8p7z\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:03.725663 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.725571 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:54:03.744166 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.744138 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:54:03.800373 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:03.800336 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:04.058812 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:04.058722 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:04.125441 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:04.125404 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z"] Oct 29 21:54:04.251080 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:04.251005 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:04.437611 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:04.437564 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:04.613694 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:04.613647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:04.614212 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:04.613980 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" event={"ID":"5da6a393-1fd5-4ab4-b273-c6d32454aa0c","Type":"ContainerStarted","Data":"20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce"} Oct 29 21:54:04.614212 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:04.614022 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" event={"ID":"5da6a393-1fd5-4ab4-b273-c6d32454aa0c","Type":"ContainerStarted","Data":"880097e6df7c5b0238ce1fad0971ead7d23d4c7ce376f781a7f6469b46c9b9f3"} Oct 29 21:54:04.628864 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:04.628827 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:54:04.659486 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:04.659435 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podStartSLOduration=1.659420458 podStartE2EDuration="1.659420458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:04.657095424 +0000 UTC m=+710.131683843" watchObservedRunningTime="2026-10-29 21:54:04.659420458 +0000 UTC m=+710.134008910" Oct 29 21:54:04.801282 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:04.801190 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:05.197930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:05.197860 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:05.369501 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:05.369458 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:06.342702 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:06.342658 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:06.522097 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:06.522062 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:06.704682 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:06.704595 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:07.217913 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:07.217868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:08.158878 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:08.158838 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:08.333654 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:08.333620 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:08.980932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:08.980859 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:09.244376 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:09.244285 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:09.417862 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:09.417816 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:10.261105 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:10.261071 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:10.410842 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:10.410802 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:54:10.453160 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:10.453119 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:10.636818 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:10.636779 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:12.019341 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.018890 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66"] Oct 29 21:54:12.023876 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.023850 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.058553 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.058512 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66"] Oct 29 21:54:12.064741 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.064717 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29c2f2\"/\"premium-w1-29c2f2-kserve-self-signed-certs\"" Oct 29 21:54:12.076109 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.076074 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-model-cache\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.076285 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.076117 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-tmp-dir\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.076285 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.076251 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jd8\" (UniqueName: \"kubernetes.io/projected/52740807-cfc4-4144-b4dc-14ef28aff6d6-kube-api-access-x8jd8\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.076285 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.076281 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-dshm\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.076460 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.076406 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52740807-cfc4-4144-b4dc-14ef28aff6d6-tls-certs\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.076460 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.076442 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-home\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.155850 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.155812 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:12.177923 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.177870 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52740807-cfc4-4144-b4dc-14ef28aff6d6-tls-certs\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.177923 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.177929 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-home\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.178185 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.177996 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-model-cache\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.178185 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.178018 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-tmp-dir\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.178185 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.178083 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jd8\" (UniqueName: \"kubernetes.io/projected/52740807-cfc4-4144-b4dc-14ef28aff6d6-kube-api-access-x8jd8\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.178185 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.178112 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-dshm\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.178719 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.178662 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-home\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.178719 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.178695 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-model-cache\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.178949 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.178753 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-tmp-dir\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.180841 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.180821 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52740807-cfc4-4144-b4dc-14ef28aff6d6-tls-certs\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.181011 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.180995 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-dshm\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.202918 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.202878 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jd8\" (UniqueName: \"kubernetes.io/projected/52740807-cfc4-4144-b4dc-14ef28aff6d6-kube-api-access-x8jd8\") pod \"premium-w1-29c2f2-kserve-57bf997fc7-48j66\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.358651 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.358609 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:12.398957 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.398919 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm"] Oct 29 21:54:12.404006 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.403977 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.423141 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.423045 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-863f1b-serving-cert\"" Oct 29 21:54:12.423141 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.423123 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 21:54:12.425106 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.423358 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 21:54:12.425106 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.423417 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b8hhr\"" Oct 29 21:54:12.436311 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.428306 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 21:54:12.436311 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.431508 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm"] Oct 29 21:54:12.480782 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.480754 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-maas-api-tls\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.480969 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.480795 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-metrics-certs\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.480969 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.480824 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.480969 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.480955 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzncp\" (UniqueName: \"kubernetes.io/projected/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-kube-api-access-mzncp\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.543181 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.543145 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:12.582053 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.582016 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzncp\" (UniqueName: \"kubernetes.io/projected/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-kube-api-access-mzncp\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.582210 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.582142 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-maas-api-tls\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.582210 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.582188 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-metrics-certs\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.582332 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.582311 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.582990 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.582969 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.584818 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.584796 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-maas-api-tls\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.584895 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.584814 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-metrics-certs\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.608445 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.608424 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzncp\" (UniqueName: \"kubernetes.io/projected/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-kube-api-access-mzncp\") pod \"maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:12.667098 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.667068 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66"] Oct 29 21:54:12.668636 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:54:12.668605 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52740807_cfc4_4144_b4dc_14ef28aff6d6.slice/crio-2227b72d9a63ed11a2029da8319fb83c4f4ea05398915ef029da25b880017726 WatchSource:0}: Error finding container 2227b72d9a63ed11a2029da8319fb83c4f4ea05398915ef029da25b880017726: Status 404 returned error can't find the container with id 2227b72d9a63ed11a2029da8319fb83c4f4ea05398915ef029da25b880017726 Oct 29 21:54:12.700328 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.700296 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" event={"ID":"52740807-cfc4-4144-b4dc-14ef28aff6d6","Type":"ContainerStarted","Data":"2227b72d9a63ed11a2029da8319fb83c4f4ea05398915ef029da25b880017726"} Oct 29 21:54:12.725301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.725268 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:12.746687 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:12.746659 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:13.066088 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:13.066060 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm"] Oct 29 21:54:13.067221 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:54:13.067195 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e22089b_9ec5_4f04_98c7_8e3a373f2dc5.slice/crio-e13f399ac08b49ae6bfb45af420037a14ea3fae4e6bc64f1f49fed8691e7acc7 WatchSource:0}: Error finding container e13f399ac08b49ae6bfb45af420037a14ea3fae4e6bc64f1f49fed8691e7acc7: Status 404 returned error can't find the container with id e13f399ac08b49ae6bfb45af420037a14ea3fae4e6bc64f1f49fed8691e7acc7 Oct 29 21:54:13.709205 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:13.709141 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" event={"ID":"52740807-cfc4-4144-b4dc-14ef28aff6d6","Type":"ContainerStarted","Data":"64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3"} Oct 29 21:54:13.711046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:13.711007 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" event={"ID":"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5","Type":"ContainerStarted","Data":"c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a"} Oct 29 21:54:13.711194 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:13.711064 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" event={"ID":"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5","Type":"ContainerStarted","Data":"e13f399ac08b49ae6bfb45af420037a14ea3fae4e6bc64f1f49fed8691e7acc7"} Oct 29 21:54:13.711194 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:13.711096 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:13.751137 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:13.750933 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podStartSLOduration=2.7509167679999997 podStartE2EDuration="2.750916768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:13.750804841 +0000 UTC m=+719.225393260" watchObservedRunningTime="2026-10-29 21:54:13.750916768 +0000 UTC m=+719.225505190" Oct 29 21:54:13.794717 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:13.794657 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" podStartSLOduration=1.398175409 podStartE2EDuration="1.794640359s" totalImagesPullingTime="396.46495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:13.793922628 +0000 UTC m=+719.268511042" watchObservedRunningTime="2026-10-29 21:54:13.794640359 +0000 UTC m=+719.269228779" Oct 29 21:54:13.801218 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:13.801189 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:13.815019 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:13.814990 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:14.224970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:14.224918 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:14.358861 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:14.358827 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:14.692584 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:14.692545 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:14.731347 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:14.731314 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:54:15.375660 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:15.375624 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:15.530623 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:54:15.530578 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:54:15.546839 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:15.546806 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:15.844938 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:15.844876 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:17.605467 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:17.605428 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:17.794539 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:17.794497 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:17.877071 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:17.876937 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:54:17.976438 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:17.976399 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:19.601487 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:19.601445 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:19.724872 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:19.724842 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 21:54:19.779663 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:19.779601 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:19.953797 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:19.953696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:21.331611 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:21.331567 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:22.359677 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:22.359645 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:22.372267 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:22.372241 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:22.650761 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:22.650671 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:22.783885 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:22.783859 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:54:22.924888 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:22.924796 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:23.221445 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.221363 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:23.391988 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.391947 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:23.586762 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.586730 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:23.725727 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.725677 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq"] Oct 29 21:54:23.731761 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.731731 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.750110 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.750080 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq"] Oct 29 21:54:23.759095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.759060 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"kube-root-ca.crt\"" Oct 29 21:54:23.759095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.759075 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"openshift-service-ca.crt\"" Oct 29 21:54:23.759095 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.759099 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"default-dockercfg-g5d5n\"" Oct 29 21:54:23.761721 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.761698 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"simulator-w5-91520c-kserve-self-signed-certs\"" Oct 29 21:54:23.816477 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.816434 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tls-certs\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.816827 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.816805 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tmp-dir\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.817039 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.817018 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rgg\" (UniqueName: \"kubernetes.io/projected/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-kube-api-access-w6rgg\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.817234 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.817218 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-home\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.817433 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.817419 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-dshm\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.817580 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.817566 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-model-cache\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.846801 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.846717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:23.918619 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.918588 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tls-certs\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.918826 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.918627 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tmp-dir\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.918826 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.918646 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rgg\" (UniqueName: \"kubernetes.io/projected/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-kube-api-access-w6rgg\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.918826 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.918696 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-home\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.918826 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.918789 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-dshm\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.918826 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.918823 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-model-cache\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.919134 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.919105 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tmp-dir\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.919192 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.919139 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-home\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.919505 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.919481 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-model-cache\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.921377 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.921353 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-dshm\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.921768 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.921744 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tls-certs\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:23.943885 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:23.943853 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rgg\" (UniqueName: \"kubernetes.io/projected/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-kube-api-access-w6rgg\") pod \"simulator-w5-91520c-kserve-55fc57dfb4-k8hcq\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:24.046353 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:24.046314 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:24.353072 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:24.353038 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq"] Oct 29 21:54:24.530326 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:24.530285 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:24.789217 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:24.789127 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" event={"ID":"4e9a296f-e4b7-45ad-a57f-3e79b29722ef","Type":"ContainerStarted","Data":"60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10"} Oct 29 21:54:24.789217 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:24.789166 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" event={"ID":"4e9a296f-e4b7-45ad-a57f-3e79b29722ef","Type":"ContainerStarted","Data":"11d1fde1b2d5cec8b0fb15fe6286608c3ac05e74e468780a15255afc22c38120"} Oct 29 21:54:24.834316 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:24.834262 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podStartSLOduration=1.834247389 podStartE2EDuration="1.834247389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:24.831588037 +0000 UTC m=+730.306176455" watchObservedRunningTime="2026-10-29 21:54:24.834247389 +0000 UTC m=+730.308835807" Oct 29 21:54:25.047297 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:25.047210 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:26.215146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:26.215111 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:26.694627 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:26.694590 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:54:27.514831 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:27.514794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:28.029313 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:28.029272 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:29.581799 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:29.581755 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:29.757888 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:29.757850 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:30.276158 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:30.276111 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:30.448393 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:30.448361 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:31.418551 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:31.418510 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:33.003146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:33.003109 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:33.990511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:33.990476 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:54:34.046723 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:34.046693 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:34.062766 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:34.062735 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:34.090454 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:34.090417 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:34.728592 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:34.728555 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:34.872533 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:34.872500 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:54:36.516392 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:36.516350 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:37.022477 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:37.022445 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:37.706255 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:37.706224 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:37.982615 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:37.982531 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:38.160227 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:38.160187 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:38.341471 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:38.341435 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:38.884992 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:38.884960 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:39.122727 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.122685 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:39.468312 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.468279 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw"] Oct 29 21:54:39.472553 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.472530 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.486586 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.486558 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw"] Oct 29 21:54:39.495029 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.495000 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"kube-root-ca.crt\"" Oct 29 21:54:39.495029 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.495000 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"openshift-service-ca.crt\"" Oct 29 21:54:39.503443 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.503414 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7e2897-30ea-4b2e-a06d-b64329faff99-tls-certs\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.503597 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.503458 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-home\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.503666 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.503624 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-model-cache\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.503728 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.503687 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-dshm\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.503823 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.503802 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-tmp-dir\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.503864 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.503855 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smhrk\" (UniqueName: \"kubernetes.io/projected/6f7e2897-30ea-4b2e-a06d-b64329faff99-kube-api-access-smhrk\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.516152 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.516128 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"default-dockercfg-x7kjc\"" Oct 29 21:54:39.516292 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.516158 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-863f1b\"/\"simulator-w6-863f1b-kserve-self-signed-certs\"" Oct 29 21:54:39.575630 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.575593 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:39.605325 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.605290 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7e2897-30ea-4b2e-a06d-b64329faff99-tls-certs\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.605325 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.605328 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-home\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.605589 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.605370 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-model-cache\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.605589 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.605404 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-dshm\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.605589 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.605457 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-tmp-dir\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.605589 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.605508 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smhrk\" (UniqueName: \"kubernetes.io/projected/6f7e2897-30ea-4b2e-a06d-b64329faff99-kube-api-access-smhrk\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.606064 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.606027 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-model-cache\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.606179 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.606136 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-tmp-dir\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.606179 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.606160 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-home\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.607975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.607955 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-dshm\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.608629 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.608609 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7e2897-30ea-4b2e-a06d-b64329faff99-tls-certs\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.632927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.630601 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smhrk\" (UniqueName: \"kubernetes.io/projected/6f7e2897-30ea-4b2e-a06d-b64329faff99-kube-api-access-smhrk\") pod \"simulator-w6-863f1b-kserve-57dd578dfc-bmcfw\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.788372 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.788279 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:39.837954 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:39.834772 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:40.130999 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:40.128132 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw"] Oct 29 21:54:40.134173 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:54:40.134141 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f7e2897_30ea_4b2e_a06d_b64329faff99.slice/crio-ebc47dfdcb7fed82dd3321809a77d74f31990120b99660d184e7ae0cddc174c1 WatchSource:0}: Error finding container ebc47dfdcb7fed82dd3321809a77d74f31990120b99660d184e7ae0cddc174c1: Status 404 returned error can't find the container with id ebc47dfdcb7fed82dd3321809a77d74f31990120b99660d184e7ae0cddc174c1 Oct 29 21:54:40.909637 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:40.909594 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" event={"ID":"6f7e2897-30ea-4b2e-a06d-b64329faff99","Type":"ContainerStarted","Data":"730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e"} Oct 29 21:54:40.909637 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:40.909642 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" event={"ID":"6f7e2897-30ea-4b2e-a06d-b64329faff99","Type":"ContainerStarted","Data":"ebc47dfdcb7fed82dd3321809a77d74f31990120b99660d184e7ae0cddc174c1"} Oct 29 21:54:40.955394 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:40.955335 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podStartSLOduration=1.955320881 podStartE2EDuration="1.955320881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:40.954380456 +0000 UTC m=+746.428968896" watchObservedRunningTime="2026-10-29 21:54:40.955320881 +0000 UTC m=+746.429909299" Oct 29 21:54:41.525729 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:41.525696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:41.795982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:41.795000 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:43.150776 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:43.150738 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:44.721773 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:44.719342 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:45.529051 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:54:45.529007 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:54:46.315779 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:46.315743 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:46.835946 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:46.834922 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:54:48.003596 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:48.003556 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:49.551969 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:49.551936 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:49.789058 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:49.789027 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:49.801572 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:49.801542 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:49.988502 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:49.988470 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:54:51.119956 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.119920 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:51.709821 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.709786 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx"] Oct 29 21:54:51.714050 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.714026 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.743629 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.743593 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-82db63\"/\"simulator-w3-82db63-kserve-self-signed-certs\"" Oct 29 21:54:51.743763 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.743745 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-82db63\"/\"default-dockercfg-dptlb\"" Oct 29 21:54:51.746206 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.746184 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx"] Oct 29 21:54:51.782201 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.782157 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-82db63\"/\"openshift-service-ca.crt\"" Oct 29 21:54:51.782201 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.782175 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-82db63\"/\"kube-root-ca.crt\"" Oct 29 21:54:51.834331 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.834291 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:51.836431 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.836388 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-tmp-dir\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.836597 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.836432 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67b8901-4c7f-4f58-a38e-6da70147e552-tls-certs\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.836597 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.836469 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-model-cache\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.836721 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.836593 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-dshm\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.836721 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.836689 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-home\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.836825 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.836735 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78jz7\" (UniqueName: \"kubernetes.io/projected/e67b8901-4c7f-4f58-a38e-6da70147e552-kube-api-access-78jz7\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.938338 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.938247 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-dshm\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.938507 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.938401 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-home\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.938507 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.938449 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78jz7\" (UniqueName: \"kubernetes.io/projected/e67b8901-4c7f-4f58-a38e-6da70147e552-kube-api-access-78jz7\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.938642 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.938575 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-tmp-dir\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.938642 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.938600 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67b8901-4c7f-4f58-a38e-6da70147e552-tls-certs\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.938642 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.938634 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-model-cache\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.938841 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.938817 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-home\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.938968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.938954 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-tmp-dir\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.939109 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.939090 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-model-cache\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.940887 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.940866 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-dshm\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.941372 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.941341 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67b8901-4c7f-4f58-a38e-6da70147e552-tls-certs\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:51.974423 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:51.972681 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78jz7\" (UniqueName: \"kubernetes.io/projected/e67b8901-4c7f-4f58-a38e-6da70147e552-kube-api-access-78jz7\") pod \"simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:52.028147 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:52.028115 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:52.337659 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:52.337624 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx"] Oct 29 21:54:52.993842 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:52.993808 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" event={"ID":"e67b8901-4c7f-4f58-a38e-6da70147e552","Type":"ContainerStarted","Data":"d0fde757f51e9dd13c83c3bccb15e593a38323e79f01cc1c753ca69142371f06"} Oct 29 21:54:52.994062 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:52.993853 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" event={"ID":"e67b8901-4c7f-4f58-a38e-6da70147e552","Type":"ContainerStarted","Data":"00055f66d9dc29e0f61b61da8b67b3cb78387baf10705935d735dae3e720c0a2"} Oct 29 21:54:53.028802 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:53.028766 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:54:53.036690 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:53.036628 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podStartSLOduration=2.036613735 podStartE2EDuration="2.036613735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:53.035661994 +0000 UTC m=+758.510250424" watchObservedRunningTime="2026-10-29 21:54:53.036613735 +0000 UTC m=+758.511202150" Oct 29 21:54:53.843177 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:53.843138 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:54.543687 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:54.543647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:56.214722 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:56.214679 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:57.086586 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:57.086505 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:57.915455 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:57.915416 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:54:59.539954 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:54:59.536856 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:00.808955 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.808919 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5"] Oct 29 21:55:00.812521 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.812488 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:00.838065 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.838038 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5"] Oct 29 21:55:00.846622 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.846594 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"premium-w5-91520c-kserve-self-signed-certs\"" Oct 29 21:55:00.928485 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.928448 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:00.948073 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.948038 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkpx\" (UniqueName: \"kubernetes.io/projected/30270cd0-e156-43f1-bab4-ce05b529dca5-kube-api-access-pjkpx\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:00.948258 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.948185 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-model-cache\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:00.948258 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.948217 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-tmp-dir\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:00.948385 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.948318 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30270cd0-e156-43f1-bab4-ce05b529dca5-tls-certs\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:00.948443 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.948385 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-dshm\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:00.948498 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:00.948438 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-home\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.051704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.050438 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-home\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.051704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.050542 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkpx\" (UniqueName: \"kubernetes.io/projected/30270cd0-e156-43f1-bab4-ce05b529dca5-kube-api-access-pjkpx\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.051704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.050650 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-model-cache\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.051704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.050684 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-tmp-dir\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.051704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.050730 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30270cd0-e156-43f1-bab4-ce05b529dca5-tls-certs\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.051704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.050789 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-dshm\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.060220 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.052599 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-tmp-dir\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.060220 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.052891 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-home\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.060220 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.055354 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-model-cache\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.061088 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.061054 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30270cd0-e156-43f1-bab4-ce05b529dca5-tls-certs\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.061219 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.061141 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-dshm\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.077485 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.077459 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkpx\" (UniqueName: \"kubernetes.io/projected/30270cd0-e156-43f1-bab4-ce05b529dca5-kube-api-access-pjkpx\") pod \"premium-w5-91520c-kserve-67bd66c8bf-xksf5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.127670 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.127607 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:01.139999 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.139965 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:01.441269 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:01.440306 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5"] Oct 29 21:55:01.445551 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:55:01.445514 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30270cd0_e156_43f1_bab4_ce05b529dca5.slice/crio-c69b6f301b821db2a89f1ab04bf9a2b1b6885f50b024f790506375d32fc4afa3 WatchSource:0}: Error finding container c69b6f301b821db2a89f1ab04bf9a2b1b6885f50b024f790506375d32fc4afa3: Status 404 returned error can't find the container with id c69b6f301b821db2a89f1ab04bf9a2b1b6885f50b024f790506375d32fc4afa3 Oct 29 21:55:02.029359 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:02.029323 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:55:02.042375 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:02.042344 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:55:02.056876 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:02.056843 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" event={"ID":"30270cd0-e156-43f1-bab4-ce05b529dca5","Type":"ContainerStarted","Data":"fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0"} Oct 29 21:55:02.057040 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:02.056890 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" event={"ID":"30270cd0-e156-43f1-bab4-ce05b529dca5","Type":"ContainerStarted","Data":"c69b6f301b821db2a89f1ab04bf9a2b1b6885f50b024f790506375d32fc4afa3"} Oct 29 21:55:02.069101 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:02.069078 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:55:02.127489 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:02.127440 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podStartSLOduration=2.127426894 podStartE2EDuration="2.127426894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:02.126538823 +0000 UTC m=+767.601127242" watchObservedRunningTime="2026-10-29 21:55:02.127426894 +0000 UTC m=+767.602015312" Oct 29 21:55:02.128646 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:02.128623 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:02.724589 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:02.724539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:04.425951 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:04.424169 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:06.017493 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:06.017460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:06.094041 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:06.094008 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:55:06.537494 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:06.537452 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:07.902825 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:07.902785 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:08.988140 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:08.988101 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:09.538953 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:09.538920 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:11.128355 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:11.128318 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:11.133728 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:11.133687 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:11.143892 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:11.143865 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:12.141594 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:12.141565 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:55:12.825664 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:12.825622 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:14.416276 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:14.416239 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:15.110031 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:15.109990 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:15.539397 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:55:15.539305 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:55:16.115724 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:16.115641 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:17.712499 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:17.712464 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:19.317044 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:19.317001 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:20.940729 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:20.940688 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:22.630457 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:22.630423 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:24.343613 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:24.343575 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:25.055854 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.055822 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk"] Oct 29 21:55:25.069567 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.069537 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.077743 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.077709 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk"] Oct 29 21:55:25.086247 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.086218 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-82db63\"/\"premium-w3-82db63-kserve-self-signed-certs\"" Oct 29 21:55:25.109694 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.109663 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-home\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.109857 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.109708 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2de66-623a-4ede-af37-c683517ca3f4-tls-certs\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.109943 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.109896 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-tmp-dir\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.110039 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.110019 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-dshm\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.110081 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.110048 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-model-cache\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.110081 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.110075 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27xxg\" (UniqueName: \"kubernetes.io/projected/b6a2de66-623a-4ede-af37-c683517ca3f4-kube-api-access-27xxg\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.167454 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.167414 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:25.211609 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.211569 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-tmp-dir\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.211798 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.211631 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-dshm\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.211798 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.211657 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-model-cache\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.211798 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.211686 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xxg\" (UniqueName: \"kubernetes.io/projected/b6a2de66-623a-4ede-af37-c683517ca3f4-kube-api-access-27xxg\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.211798 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.211765 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-home\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.211798 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.211792 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2de66-623a-4ede-af37-c683517ca3f4-tls-certs\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.212261 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.212235 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-tmp-dir\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.212429 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.212399 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-home\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.212533 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.212333 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-model-cache\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.214525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.214502 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2de66-623a-4ede-af37-c683517ca3f4-tls-certs\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.214771 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.214750 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-dshm\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.252016 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.251925 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xxg\" (UniqueName: \"kubernetes.io/projected/b6a2de66-623a-4ede-af37-c683517ca3f4-kube-api-access-27xxg\") pod \"premium-w3-82db63-kserve-5fcc59f7f8-wqlqk\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.384215 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.384180 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:25.692322 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.692289 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk"] Oct 29 21:55:25.693393 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:55:25.693365 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6a2de66_623a_4ede_af37_c683517ca3f4.slice/crio-1b90d8885ca2f297b3a01f23f81f72e95df841c604758382ff220b25575e7a4f WatchSource:0}: Error finding container 1b90d8885ca2f297b3a01f23f81f72e95df841c604758382ff220b25575e7a4f: Status 404 returned error can't find the container with id 1b90d8885ca2f297b3a01f23f81f72e95df841c604758382ff220b25575e7a4f Oct 29 21:55:25.976565 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:25.976473 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:55:26.156942 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:26.155982 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:26.250798 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:26.250714 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" event={"ID":"b6a2de66-623a-4ede-af37-c683517ca3f4","Type":"ContainerStarted","Data":"74ea3a1ac3c74a4c2a6e4d3baaba1818578526215ac1c09b44b4434c5d219389"} Oct 29 21:55:26.250798 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:26.250755 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" event={"ID":"b6a2de66-623a-4ede-af37-c683517ca3f4","Type":"ContainerStarted","Data":"1b90d8885ca2f297b3a01f23f81f72e95df841c604758382ff220b25575e7a4f"} Oct 29 21:55:26.298451 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:26.298388 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podStartSLOduration=1.298369881 podStartE2EDuration="1.298369881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:26.292375849 +0000 UTC m=+791.766964300" watchObservedRunningTime="2026-10-29 21:55:26.298369881 +0000 UTC m=+791.772958301" Oct 29 21:55:26.385047 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:26.385012 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:27.964944 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:27.960697 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:29.799303 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.799267 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:29.869581 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.869545 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65"] Oct 29 21:55:29.873674 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.873645 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:29.893973 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.893941 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65"] Oct 29 21:55:29.902175 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.902146 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"kube-root-ca.crt\"" Oct 29 21:55:29.902293 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.902218 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"premium-w4-cea94a-kserve-self-signed-certs\"" Oct 29 21:55:29.902293 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.902227 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"openshift-service-ca.crt\"" Oct 29 21:55:29.902293 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.902243 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cea94a\"/\"default-dockercfg-s86jj\"" Oct 29 21:55:29.990069 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.990039 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-dshm\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:29.990253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.990091 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-model-cache\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:29.990253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.990180 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a012f50f-b410-4b94-96bb-1ea578bc090b-tls-certs\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:29.990357 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.990289 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-home\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:29.990357 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.990317 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-tmp-dir\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:29.990468 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.990433 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtf8\" (UniqueName: \"kubernetes.io/projected/a012f50f-b410-4b94-96bb-1ea578bc090b-kube-api-access-7jtf8\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:29.994704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:29.994674 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:30.091727 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.091683 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-home\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.091727 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.091731 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-tmp-dir\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.092020 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.091785 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtf8\" (UniqueName: \"kubernetes.io/projected/a012f50f-b410-4b94-96bb-1ea578bc090b-kube-api-access-7jtf8\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.092020 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.091876 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-dshm\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.092020 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.091967 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-model-cache\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.092020 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.092002 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a012f50f-b410-4b94-96bb-1ea578bc090b-tls-certs\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.092968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.092919 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-home\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.093245 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.093220 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-model-cache\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.093346 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.093273 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-tmp-dir\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.094653 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.094634 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-dshm\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.095289 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.095268 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a012f50f-b410-4b94-96bb-1ea578bc090b-tls-certs\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.118939 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.116682 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtf8\" (UniqueName: \"kubernetes.io/projected/a012f50f-b410-4b94-96bb-1ea578bc090b-kube-api-access-7jtf8\") pod \"premium-w4-cea94a-kserve-59cc96949c-nvq65\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.188107 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.188070 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:30.492292 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:30.492260 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65"] Oct 29 21:55:30.493432 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:55:30.493404 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda012f50f_b410_4b94_96bb_1ea578bc090b.slice/crio-58aea1f27b7cc4844a34da365c4118230d0a702d889eed95a8ff85d656b22425 WatchSource:0}: Error finding container 58aea1f27b7cc4844a34da365c4118230d0a702d889eed95a8ff85d656b22425: Status 404 returned error can't find the container with id 58aea1f27b7cc4844a34da365c4118230d0a702d889eed95a8ff85d656b22425 Oct 29 21:55:31.307341 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.307300 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" event={"ID":"a012f50f-b410-4b94-96bb-1ea578bc090b","Type":"ContainerStarted","Data":"b960d756db066895802d99883bac4c90296403f82d4d4d48c4f4383e98ea6aff"} Oct 29 21:55:31.307341 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.307347 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" event={"ID":"a012f50f-b410-4b94-96bb-1ea578bc090b","Type":"ContainerStarted","Data":"58aea1f27b7cc4844a34da365c4118230d0a702d889eed95a8ff85d656b22425"} Oct 29 21:55:31.353193 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.353142 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podStartSLOduration=2.353127623 podStartE2EDuration="2.353127623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:31.351942451 +0000 UTC m=+796.826530869" watchObservedRunningTime="2026-10-29 21:55:31.353127623 +0000 UTC m=+796.827716064" Oct 29 21:55:31.400244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.399899 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4"] Oct 29 21:55:31.403885 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.403862 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.427638 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.427603 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4"] Oct 29 21:55:31.471339 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.471318 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-29c2f2-openshift-default-dockercfg-t47zd\"" Oct 29 21:55:31.505246 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.505218 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.505397 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.505257 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.505397 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.505296 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.505488 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.505386 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5twm\" (UniqueName: \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-kube-api-access-l5twm\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.505526 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.505500 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92724a9e-daa1-4839-b12e-38db23b6f8be-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.505570 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.505556 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.505618 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.505597 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.505708 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.505692 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.505750 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.505718 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.545685 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.545651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:31.607146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607111 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92724a9e-daa1-4839-b12e-38db23b6f8be-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607333 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607165 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607333 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607192 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607333 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607236 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607496 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607353 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607496 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607453 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607701 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607679 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607788 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607757 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607842 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607788 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607899 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607848 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607899 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607851 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.607899 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.607867 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5twm\" (UniqueName: \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-kube-api-access-l5twm\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.608223 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.608203 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.608464 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.608441 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92724a9e-daa1-4839-b12e-38db23b6f8be-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.609932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.609887 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.610041 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.609957 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.632274 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.632249 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.632403 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.632254 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5twm\" (UniqueName: \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-kube-api-access-l5twm\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:31.720056 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:31.720023 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:32.049958 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:32.049858 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4"] Oct 29 21:55:32.051593 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:55:32.051563 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92724a9e_daa1_4839_b12e_38db23b6f8be.slice/crio-4744e8b57b69c22c13ab431f786d538bda891f3519eba13639e623bf0ad5d76c WatchSource:0}: Error finding container 4744e8b57b69c22c13ab431f786d538bda891f3519eba13639e623bf0ad5d76c: Status 404 returned error can't find the container with id 4744e8b57b69c22c13ab431f786d538bda891f3519eba13639e623bf0ad5d76c Oct 29 21:55:32.054419 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:32.054385 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:55:32.054510 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:32.054463 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:55:32.054563 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:32.054506 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:55:32.188396 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:32.188337 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:32.336970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:32.336942 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" event={"ID":"92724a9e-daa1-4839-b12e-38db23b6f8be","Type":"ContainerStarted","Data":"b6decfa05bdd76780ee0b17ad6b1531ba32ebfce13e326723d4791d4c0aba981"} Oct 29 21:55:32.337511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:32.337488 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" event={"ID":"92724a9e-daa1-4839-b12e-38db23b6f8be","Type":"ContainerStarted","Data":"4744e8b57b69c22c13ab431f786d538bda891f3519eba13639e623bf0ad5d76c"} Oct 29 21:55:32.384459 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:32.384408 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" podStartSLOduration=1.384394947 podStartE2EDuration="1.384394947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:32.382143023 +0000 UTC m=+797.856731441" watchObservedRunningTime="2026-10-29 21:55:32.384394947 +0000 UTC m=+797.858983365" Oct 29 21:55:32.720449 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:32.720367 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:33.258543 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:33.258505 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:33.725050 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:33.725009 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:55:33.725646 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:33.725612 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 21:55:34.725926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:34.721116 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:55:34.725926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:34.721172 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 21:55:35.072889 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:35.072794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:35.385108 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:35.385077 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:35.397887 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:35.397863 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:35.720417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:35.720334 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:55:35.720568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:35.720410 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 21:55:36.391304 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:36.391271 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:55:36.720874 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:36.720774 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:55:36.720874 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:36.720851 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 21:55:36.982191 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:36.982051 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:37.720950 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:37.720878 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:55:37.721449 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:37.720972 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 21:55:38.722918 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:38.721746 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 29 21:55:38.722918 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:38.721808 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 29 21:55:38.856517 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:38.854710 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:40.076836 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:40.076806 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:40.077351 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:40.077114 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:40.078123 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:40.078103 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 21:55:40.191319 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:40.188889 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:40.208154 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:40.208123 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:40.422166 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:40.422074 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:55:40.708754 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:40.708665 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:42.534249 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:42.534207 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:44.340225 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:44.340187 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:45.528059 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:55:45.528023 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:55:46.033035 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:46.032998 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:47.833484 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:47.833448 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:49.588743 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:49.588707 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:50.996619 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:50.996579 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:55:51.243929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:51.243338 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:53.100802 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:53.100755 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:54.833809 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:54.833735 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:56.519980 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:56.519941 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:58.159842 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:58.159798 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:58.334595 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:58.334557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:55:59.928725 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:55:59.928675 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:01.673134 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:01.673084 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:03.529690 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:03.529634 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:05.246472 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:05.246430 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:06.577974 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:06.577940 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:06.854758 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:06.854674 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:08.646942 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:08.646882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:10.339029 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:10.338993 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:12.114129 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:12.114073 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:13.841936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:13.841871 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:14.556197 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:14.556147 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:14.579073 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:14.579037 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:56:15.537223 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:56:15.537189 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:56:15.747567 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:15.747522 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:17.425046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:17.424974 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:18.796094 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.796058 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2"] Oct 29 21:56:18.801456 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.801430 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:18.819196 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.819160 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2"] Oct 29 21:56:18.827949 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.827866 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-92d984\"/\"default-dockercfg-fkmhr\"" Oct 29 21:56:18.827949 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.827866 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-92d984\"/\"openshift-service-ca.crt\"" Oct 29 21:56:18.828190 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.827973 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-92d984\"/\"distinct-w2-92d984-kserve-self-signed-certs\"" Oct 29 21:56:18.828190 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.828078 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-92d984\"/\"kube-root-ca.crt\"" Oct 29 21:56:18.954594 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.954548 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-model-cache\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:18.954794 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.954616 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-home\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:18.954794 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.954736 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-dshm\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:18.954954 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.954798 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rc5q\" (UniqueName: \"kubernetes.io/projected/5e982e38-3019-4f3b-ae7d-a7273433a984-kube-api-access-7rc5q\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:18.955025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.955003 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-tmp-dir\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:18.955120 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.955102 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e982e38-3019-4f3b-ae7d-a7273433a984-tls-certs\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:18.956551 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:18.956522 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:19.055788 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.055686 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-tmp-dir\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.055986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.055794 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e982e38-3019-4f3b-ae7d-a7273433a984-tls-certs\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.055986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.055858 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-model-cache\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.055986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.055899 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-home\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.055986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.055938 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-dshm\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.055986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.055966 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rc5q\" (UniqueName: \"kubernetes.io/projected/5e982e38-3019-4f3b-ae7d-a7273433a984-kube-api-access-7rc5q\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.056690 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.056662 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-home\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.056932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.056714 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-tmp-dir\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.057068 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.057043 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-model-cache\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.062494 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.062470 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e982e38-3019-4f3b-ae7d-a7273433a984-tls-certs\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.070545 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.070514 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-dshm\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.089930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.088012 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rc5q\" (UniqueName: \"kubernetes.io/projected/5e982e38-3019-4f3b-ae7d-a7273433a984-kube-api-access-7rc5q\") pod \"distinct-w2-92d984-kserve-7d48cb8547-6zcs2\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.117804 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.117770 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:19.207296 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.206583 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:19.459410 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.459372 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2"] Oct 29 21:56:19.463061 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:56:19.463030 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e982e38_3019_4f3b_ae7d_a7273433a984.slice/crio-f732a5535aab950b3b229520db2831ad91feda0d0868bfb56f17363846bc8ffe WatchSource:0}: Error finding container f732a5535aab950b3b229520db2831ad91feda0d0868bfb56f17363846bc8ffe: Status 404 returned error can't find the container with id f732a5535aab950b3b229520db2831ad91feda0d0868bfb56f17363846bc8ffe Oct 29 21:56:19.694432 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.694392 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" event={"ID":"5e982e38-3019-4f3b-ae7d-a7273433a984","Type":"ContainerStarted","Data":"e55c013e0b8477cccf5128a8eb900bafb619721718f971648bccc84bc595c6fb"} Oct 29 21:56:19.694643 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.694442 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" event={"ID":"5e982e38-3019-4f3b-ae7d-a7273433a984","Type":"ContainerStarted","Data":"f732a5535aab950b3b229520db2831ad91feda0d0868bfb56f17363846bc8ffe"} Oct 29 21:56:19.738768 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:19.738666 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podStartSLOduration=1.738650007 podStartE2EDuration="1.738650007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:19.73578697 +0000 UTC m=+845.210375412" watchObservedRunningTime="2026-10-29 21:56:19.738650007 +0000 UTC m=+845.213238424" Oct 29 21:56:20.117921 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:20.117883 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:20.938579 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:20.938538 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:22.650160 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:22.650116 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:24.557474 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:24.557433 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:26.136279 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:26.136188 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:27.845072 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:27.845035 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:29.118975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:29.118932 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:29.143111 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:29.143077 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:29.735011 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:29.734977 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:29.793422 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:29.793394 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:56:31.537071 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:31.537029 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:33.225320 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:33.225272 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:34.299417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.299378 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c"] Oct 29 21:56:34.303767 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.303744 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.330788 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.330760 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-91520c\"/\"e2e-conflict-b-fb842e5c-kserve-self-signed-certs\"" Oct 29 21:56:34.333598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.333573 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c"] Oct 29 21:56:34.408996 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.408953 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:34.432966 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.432939 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-home\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.433146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.433006 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/371c3027-1c25-4574-9483-b7c3ed5e27f8-tls-certs\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.433497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.433164 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f76\" (UniqueName: \"kubernetes.io/projected/371c3027-1c25-4574-9483-b7c3ed5e27f8-kube-api-access-r4f76\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.433497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.433239 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-dshm\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.433497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.433356 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-tmp-dir\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.433497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.433390 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-model-cache\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.534439 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.534395 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-tmp-dir\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.534656 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.534461 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-model-cache\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.534656 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.534511 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-home\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.534656 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.534575 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/371c3027-1c25-4574-9483-b7c3ed5e27f8-tls-certs\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.534656 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.534638 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f76\" (UniqueName: \"kubernetes.io/projected/371c3027-1c25-4574-9483-b7c3ed5e27f8-kube-api-access-r4f76\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.534883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.534678 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-dshm\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.534988 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.534964 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-model-cache\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.535046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.534976 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-tmp-dir\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.535370 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.535348 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-home\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.537368 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.537338 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-dshm\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.537475 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.537431 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/371c3027-1c25-4574-9483-b7c3ed5e27f8-tls-certs\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.559527 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.559450 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f76\" (UniqueName: \"kubernetes.io/projected/371c3027-1c25-4574-9483-b7c3ed5e27f8-kube-api-access-r4f76\") pod \"e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.619725 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.619687 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:34.941284 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:34.941252 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c"] Oct 29 21:56:34.942662 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:56:34.942640 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod371c3027_1c25_4574_9483_b7c3ed5e27f8.slice/crio-10feb1539095172c4630264c120e658977e6e20c84f68b6ffd30fbc377e67d43 WatchSource:0}: Error finding container 10feb1539095172c4630264c120e658977e6e20c84f68b6ffd30fbc377e67d43: Status 404 returned error can't find the container with id 10feb1539095172c4630264c120e658977e6e20c84f68b6ffd30fbc377e67d43 Oct 29 21:56:35.030697 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:35.030647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:35.822045 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:35.822006 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" event={"ID":"371c3027-1c25-4574-9483-b7c3ed5e27f8","Type":"ContainerStarted","Data":"a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff"} Oct 29 21:56:35.822045 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:35.822051 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" event={"ID":"371c3027-1c25-4574-9483-b7c3ed5e27f8","Type":"ContainerStarted","Data":"10feb1539095172c4630264c120e658977e6e20c84f68b6ffd30fbc377e67d43"} Oct 29 21:56:35.868150 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:35.868087 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" podStartSLOduration=1.8680669189999999 podStartE2EDuration="1.868066919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:35.863654488 +0000 UTC m=+861.338242906" watchObservedRunningTime="2026-10-29 21:56:35.868066919 +0000 UTC m=+861.342655341" Oct 29 21:56:36.621000 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:36.620970 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:36.726001 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:36.725944 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:38.405327 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:38.405275 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:40.103020 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:40.102978 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:40.623053 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:40.623010 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:56:41.722008 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:41.721969 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:43.458962 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:43.458894 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:44.621150 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:44.621116 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:44.635068 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:44.635039 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:44.898308 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:44.898225 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:56:45.132506 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:45.132476 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:45.527015 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:56:45.526976 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:56:46.542865 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.542823 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp"] Oct 29 21:56:46.547231 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.547210 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.563145 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.563124 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-91520c-openshift-default-dockercfg-gg8xj\"" Oct 29 21:56:46.578280 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.578256 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp"] Oct 29 21:56:46.686934 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.686872 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.687116 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.686941 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxbvh\" (UniqueName: \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-kube-api-access-wxbvh\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.687116 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.687034 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.687116 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.687081 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.687240 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.687137 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.687240 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.687177 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.687316 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.687284 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.687362 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.687320 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.687435 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.687412 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.788575 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.788536 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.788774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.788610 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.788774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.788654 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.788774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.788722 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.788774 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.788742 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.789065 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.788781 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.789065 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.788816 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.789065 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.788841 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxbvh\" (UniqueName: \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-kube-api-access-wxbvh\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.789065 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.788868 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.789065 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.789016 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.789322 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.789274 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.789879 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.789852 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.790019 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.789989 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.790637 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.790597 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.791743 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.791720 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.793223 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.793160 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.814724 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.814693 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.814967 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.814895 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxbvh\" (UniqueName: \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-kube-api-access-wxbvh\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.861508 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.861478 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:46.864533 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:46.864493 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:47.188832 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:47.188794 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp"] Oct 29 21:56:47.189690 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:56:47.189666 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d14e43_1bb1_43cd_87d2_6a7f31c0523d.slice/crio-6763362f66e99f470e7b1c8e737a1969b319a3a2fb3e6ffb9b0f7f5e84ec0e57 WatchSource:0}: Error finding container 6763362f66e99f470e7b1c8e737a1969b319a3a2fb3e6ffb9b0f7f5e84ec0e57: Status 404 returned error can't find the container with id 6763362f66e99f470e7b1c8e737a1969b319a3a2fb3e6ffb9b0f7f5e84ec0e57 Oct 29 21:56:47.192218 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:47.192175 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:56:47.192299 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:47.192244 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:56:47.192299 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:47.192276 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:56:47.910988 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:47.910954 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" event={"ID":"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d","Type":"ContainerStarted","Data":"cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2"} Oct 29 21:56:47.911461 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:47.911010 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" event={"ID":"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d","Type":"ContainerStarted","Data":"6763362f66e99f470e7b1c8e737a1969b319a3a2fb3e6ffb9b0f7f5e84ec0e57"} Oct 29 21:56:47.970806 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:47.970742 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" podStartSLOduration=1.970722509 podStartE2EDuration="1.970722509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:47.969181375 +0000 UTC m=+873.443769820" watchObservedRunningTime="2026-10-29 21:56:47.970722509 +0000 UTC m=+873.445310928" Oct 29 21:56:48.516829 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:48.516797 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:48.690613 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:48.690572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:48.862602 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:48.862569 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:48.864303 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:48.864275 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:48.864444 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:48.864326 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 21:56:49.862524 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:49.862484 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:49.862966 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:49.862548 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 21:56:50.468198 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:50.468155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:50.862048 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:50.862012 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:50.862238 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:50.862068 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 21:56:51.862927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:51.862881 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:51.863420 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:51.862943 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 21:56:52.343537 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:52.343398 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:52.862586 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:52.862550 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:52.862806 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:52.862612 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 21:56:53.012404 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:53.012360 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:53.862432 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:53.862361 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 29 21:56:53.862432 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:53.862420 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 29 21:56:54.152931 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.152813 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:54.515460 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.515372 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6"] Oct 29 21:56:54.519811 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.519786 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.536966 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.536933 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-92d984-openshift-default-dockercfg-5jl7t\"" Oct 29 21:56:54.556160 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.556123 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6"] Oct 29 21:56:54.602135 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.602101 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.602135 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.602139 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.602369 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.602168 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjgc6\" (UniqueName: \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-kube-api-access-fjgc6\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.602369 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.602268 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e290a6d-f30b-4005-b567-94091995ec44-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.602369 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.602307 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.602529 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.602432 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.602529 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.602481 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.602593 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.602531 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e290a6d-f30b-4005-b567-94091995ec44-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.602593 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.602577 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704061 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704030 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704061 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704064 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704287 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704089 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgc6\" (UniqueName: \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-kube-api-access-fjgc6\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704287 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704117 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e290a6d-f30b-4005-b567-94091995ec44-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704287 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704171 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704287 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704243 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704287 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704273 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704554 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704307 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e290a6d-f30b-4005-b567-94091995ec44-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704554 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704376 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704554 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704459 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704554 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704512 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704619 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.704770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.704707 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.705189 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.705169 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e290a6d-f30b-4005-b567-94091995ec44-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.706704 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.706681 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.706840 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.706824 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e290a6d-f30b-4005-b567-94091995ec44-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.728745 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.728716 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.729117 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.729096 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjgc6\" (UniqueName: \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-kube-api-access-fjgc6\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.839881 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.839849 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:54.869448 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.869412 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:54.870691 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.870669 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:54.871394 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:54.871365 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 21:56:55.170681 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:55.170655 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6"] Oct 29 21:56:55.172129 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:56:55.172099 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e290a6d_f30b_4005_b567_94091995ec44.slice/crio-f2378b23d30edd6347a368df38547a75aacef4dbd4bdcd24822eaccacdb2618a WatchSource:0}: Error finding container f2378b23d30edd6347a368df38547a75aacef4dbd4bdcd24822eaccacdb2618a: Status 404 returned error can't find the container with id f2378b23d30edd6347a368df38547a75aacef4dbd4bdcd24822eaccacdb2618a Oct 29 21:56:55.174528 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:55.174497 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:56:55.174620 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:55.174558 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:56:55.174620 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:55.174586 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:56:55.927930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:55.927446 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:56.000399 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:56.000341 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" event={"ID":"9e290a6d-f30b-4005-b567-94091995ec44","Type":"ContainerStarted","Data":"5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5"} Oct 29 21:56:56.000629 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:56.000410 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" event={"ID":"9e290a6d-f30b-4005-b567-94091995ec44","Type":"ContainerStarted","Data":"f2378b23d30edd6347a368df38547a75aacef4dbd4bdcd24822eaccacdb2618a"} Oct 29 21:56:56.144930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:56.143866 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:56.843930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:56.840968 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:56:56.843930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:56.843015 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:56:56.843930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:56.843074 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 21:56:57.841742 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:57.841269 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:56:57.841742 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:57.841364 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 21:56:57.899984 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:57.899927 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:56:58.840926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:58.840362 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:56:58.840926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:58.840467 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 21:56:59.840928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:59.840245 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:56:59.840928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:59.840305 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 21:56:59.876982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:56:59.876947 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:00.840309 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:00.840257 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:57:00.840618 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:00.840576 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 21:57:01.653955 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:01.653882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:01.840682 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:01.840637 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 29 21:57:01.840883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:01.840711 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 29 21:57:01.850408 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:01.850366 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:02.845993 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:02.845959 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:57:02.846473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:02.846351 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:57:02.847156 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:02.847132 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 21:57:02.908332 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:02.908265 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" podStartSLOduration=8.908246067 podStartE2EDuration="8.908246067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:56.06844732 +0000 UTC m=+881.543035737" watchObservedRunningTime="2026-10-29 21:57:02.908246067 +0000 UTC m=+888.382834490" Oct 29 21:57:03.346592 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:03.346551 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:05.051151 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.051107 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:05.765289 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.765250 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2"] Oct 29 21:57:05.770162 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.770134 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:05.789438 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.789407 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2"] Oct 29 21:57:05.796541 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.796497 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-92d984\"/\"distinct-2-w2-92d984-kserve-self-signed-certs\"" Oct 29 21:57:05.874315 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.874277 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:05.911366 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.911333 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-dshm\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:05.911542 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.911377 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-home\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:05.911542 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.911487 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06fa9242-c768-443f-9a04-5ac39146babf-tls-certs\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:05.911542 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.911522 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-model-cache\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:05.911682 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.911560 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rct\" (UniqueName: \"kubernetes.io/projected/06fa9242-c768-443f-9a04-5ac39146babf-kube-api-access-w9rct\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:05.911682 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:05.911661 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-tmp-dir\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.013297 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.013256 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-dshm\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.013495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.013308 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-home\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.013495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.013345 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06fa9242-c768-443f-9a04-5ac39146babf-tls-certs\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.013495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.013379 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-model-cache\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.013495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.013412 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rct\" (UniqueName: \"kubernetes.io/projected/06fa9242-c768-443f-9a04-5ac39146babf-kube-api-access-w9rct\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.013495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.013484 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-tmp-dir\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.013960 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.013932 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-model-cache\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.013960 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.013949 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-tmp-dir\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.014131 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.014093 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-home\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.016053 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.015998 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-dshm\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.016320 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.016301 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06fa9242-c768-443f-9a04-5ac39146babf-tls-certs\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.039049 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.039017 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rct\" (UniqueName: \"kubernetes.io/projected/06fa9242-c768-443f-9a04-5ac39146babf-kube-api-access-w9rct\") pod \"distinct-2-w2-92d984-kserve-5897df59d8-dkdl2\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.093253 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.093220 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:06.413673 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.413634 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2"] Oct 29 21:57:06.414423 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:57:06.414390 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06fa9242_c768_443f_9a04_5ac39146babf.slice/crio-4346243403ac410a806407fb8133b3e1d6d9a6aaa4c664727ff0c213e85f4729 WatchSource:0}: Error finding container 4346243403ac410a806407fb8133b3e1d6d9a6aaa4c664727ff0c213e85f4729: Status 404 returned error can't find the container with id 4346243403ac410a806407fb8133b3e1d6d9a6aaa4c664727ff0c213e85f4729 Oct 29 21:57:06.710429 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.710346 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:06.886464 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:06.886433 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:07.108584 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:07.108554 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" event={"ID":"06fa9242-c768-443f-9a04-5ac39146babf","Type":"ContainerStarted","Data":"5065a022f6898273d2a05456a40366f9aaeb3c3a48e7c5da8d1d9c195614dd24"} Oct 29 21:57:07.109113 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:07.108594 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" event={"ID":"06fa9242-c768-443f-9a04-5ac39146babf","Type":"ContainerStarted","Data":"4346243403ac410a806407fb8133b3e1d6d9a6aaa4c664727ff0c213e85f4729"} Oct 29 21:57:07.151345 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:07.151285 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podStartSLOduration=2.151269311 podStartE2EDuration="2.151269311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:07.150304518 +0000 UTC m=+892.624892936" watchObservedRunningTime="2026-10-29 21:57:07.151269311 +0000 UTC m=+892.625857731" Oct 29 21:57:08.093799 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:08.093768 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:08.542451 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:08.542333 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:10.088544 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:10.088511 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:57:10.209203 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:10.208668 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:12.089924 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:12.089538 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:13.844101 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:13.844067 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:14.936259 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:14.936218 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 21:57:14.936744 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:14.936343 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 21:57:15.530124 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:57:15.530087 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:57:15.543701 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:15.543669 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:16.094185 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:16.094153 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:16.107572 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:16.107537 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:16.225686 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:16.225655 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:17.250139 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:17.250089 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:18.877684 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:18.877634 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:20.551486 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:20.551453 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c"] Oct 29 21:57:20.555333 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:20.555263 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:20.736495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:20.736463 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:21.115620 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:21.115583 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:22.232456 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:22.232418 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:23.921439 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:23.921401 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:25.734712 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:25.734673 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:27.241387 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:27.241353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:27.429929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:27.426468 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:29.245376 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:29.245338 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:31.087003 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:31.085624 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:31.941579 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:31.941538 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:32.127185 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:32.127143 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:32.845270 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:32.845230 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:34.456393 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:34.456357 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:35.593296 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:35.593228 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" podUID="371c3027-1c25-4574-9483-b7c3ed5e27f8" containerName="main" containerID="cri-o://a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff" gracePeriod=45 Oct 29 21:57:35.888738 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:35.888708 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:57:36.004347 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.004317 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-home\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-home\") pod \"371c3027-1c25-4574-9483-b7c3ed5e27f8\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " Oct 29 21:57:36.004506 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.004390 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-tmp-dir\") pod \"371c3027-1c25-4574-9483-b7c3ed5e27f8\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " Oct 29 21:57:36.004549 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.004524 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371c3027-1c25-4574-9483-b7c3ed5e27f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/371c3027-1c25-4574-9483-b7c3ed5e27f8-tls-certs\") pod \"371c3027-1c25-4574-9483-b7c3ed5e27f8\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " Oct 29 21:57:36.004670 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.004634 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-tmp-dir" pod "371c3027-1c25-4574-9483-b7c3ed5e27f8" (UID: "371c3027-1c25-4574-9483-b7c3ed5e27f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:57:36.004670 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.004642 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-dshm\") pod \"371c3027-1c25-4574-9483-b7c3ed5e27f8\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " Oct 29 21:57:36.004883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.004691 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-model-cache\") pod \"371c3027-1c25-4574-9483-b7c3ed5e27f8\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " Oct 29 21:57:36.004883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.004641 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-home" pod "371c3027-1c25-4574-9483-b7c3ed5e27f8" (UID: "371c3027-1c25-4574-9483-b7c3ed5e27f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:57:36.004883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.004730 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371c3027-1c25-4574-9483-b7c3ed5e27f8-kube-api-access-r4f76\" (UniqueName: \"kubernetes.io/projected/371c3027-1c25-4574-9483-b7c3ed5e27f8-kube-api-access-r4f76\") pod \"371c3027-1c25-4574-9483-b7c3ed5e27f8\" (UID: \"371c3027-1c25-4574-9483-b7c3ed5e27f8\") " Oct 29 21:57:36.005073 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.004944 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-model-cache" pod "371c3027-1c25-4574-9483-b7c3ed5e27f8" (UID: "371c3027-1c25-4574-9483-b7c3ed5e27f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:57:36.005183 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.005167 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:57:36.005237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.005185 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:57:36.005237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.005214 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:57:36.006895 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.006873 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-dshm" pod "371c3027-1c25-4574-9483-b7c3ed5e27f8" (UID: "371c3027-1c25-4574-9483-b7c3ed5e27f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 21:57:36.006895 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.006891 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371c3027-1c25-4574-9483-b7c3ed5e27f8-kube-api-access-r4f76" pod "371c3027-1c25-4574-9483-b7c3ed5e27f8" (UID: "371c3027-1c25-4574-9483-b7c3ed5e27f8"). InnerVolumeSpecName "kube-api-access-r4f76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 21:57:36.007043 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.006893 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371c3027-1c25-4574-9483-b7c3ed5e27f8-tls-certs" pod "371c3027-1c25-4574-9483-b7c3ed5e27f8" (UID: "371c3027-1c25-4574-9483-b7c3ed5e27f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 21:57:36.106726 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.106695 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/371c3027-1c25-4574-9483-b7c3ed5e27f8-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:57:36.106726 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.106722 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/371c3027-1c25-4574-9483-b7c3ed5e27f8-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:57:36.106726 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.106731 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4f76\" (UniqueName: \"kubernetes.io/projected/371c3027-1c25-4574-9483-b7c3ed5e27f8-kube-api-access-r4f76\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 21:57:36.236060 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.235969 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:36.373234 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.373192 4952 generic.go:356] "Generic (PLEG): container finished" podID="371c3027-1c25-4574-9483-b7c3ed5e27f8" containerID="a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff" exitCode=0 Oct 29 21:57:36.373390 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.373290 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" Oct 29 21:57:36.373390 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.373325 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" event={"ID":"371c3027-1c25-4574-9483-b7c3ed5e27f8","Type":"ContainerDied","Data":"a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff"} Oct 29 21:57:36.373390 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.373369 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c" event={"ID":"371c3027-1c25-4574-9483-b7c3ed5e27f8","Type":"ContainerDied","Data":"10feb1539095172c4630264c120e658977e6e20c84f68b6ffd30fbc377e67d43"} Oct 29 21:57:36.373390 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.373387 4952 scope.go:122] "RemoveContainer" containerID="a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff" Oct 29 21:57:36.384973 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.384948 4952 scope.go:122] "RemoveContainer" containerID="a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff" Oct 29 21:57:36.385289 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:57:36.385268 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff\": container with ID starting with a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff not found: ID does not exist" containerID="a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff" Oct 29 21:57:36.385361 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.385305 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff"} err="failed to get container status \"a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff\": rpc error: code = NotFound desc = could not find container \"a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff\": container with ID starting with a52fcc8b6dab08c8df05f77f5bc0868012fe4244feeedafff428975815581aff not found: ID does not exist" Oct 29 21:57:36.427981 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.427952 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c"] Oct 29 21:57:36.432872 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.432839 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/e2e-conflict-b-fb842e5c-kserve-84cbfd7b46-7q29c"] Oct 29 21:57:36.644090 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:36.644055 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371c3027-1c25-4574-9483-b7c3ed5e27f8" path="/var/lib/kubelet/pods/371c3027-1c25-4574-9483-b7c3ed5e27f8/volumes" Oct 29 21:57:37.910967 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:37.910929 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:39.505516 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:39.505474 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:39.703728 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:39.703686 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:41.233898 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.233860 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:41.871995 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.871957 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7"] Oct 29 21:57:41.872769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.872730 4952 cpu_manager.go:414] "containerMap: removing container" podUID="371c3027-1c25-4574-9483-b7c3ed5e27f8" containerName="main" Oct 29 21:57:41.872769 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.872752 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371c3027-1c25-4574-9483-b7c3ed5e27f8" containerName="main" Oct 29 21:57:41.873014 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.872890 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="371c3027-1c25-4574-9483-b7c3ed5e27f8" containerName="main" Oct 29 21:57:41.880503 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.880453 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:41.896513 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.896468 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7"] Oct 29 21:57:41.903772 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.903581 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-82db63\"/\"distinct-2-w3-82db63-kserve-self-signed-certs\"" Oct 29 21:57:41.976982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.976943 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-tmp-dir\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:41.977188 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.977002 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-model-cache\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:41.977188 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.977157 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9tl\" (UniqueName: \"kubernetes.io/projected/068d660a-401c-4b0a-9717-3a162259439a-kube-api-access-ld9tl\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:41.977324 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.977224 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-dshm\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:41.977400 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.977367 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-home\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:41.977448 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.977399 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068d660a-401c-4b0a-9717-3a162259439a-tls-certs\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:41.999142 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:41.999099 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:42.081879 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.081554 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-tmp-dir\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.081879 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.081608 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-model-cache\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.081879 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.081670 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9tl\" (UniqueName: \"kubernetes.io/projected/068d660a-401c-4b0a-9717-3a162259439a-kube-api-access-ld9tl\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.081879 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.081736 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-dshm\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.081879 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.081832 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-home\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.081879 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.081856 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068d660a-401c-4b0a-9717-3a162259439a-tls-certs\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.082966 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.082938 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-tmp-dir\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.083372 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.083352 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-model-cache\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.083740 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.083721 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-home\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.088018 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.087183 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-dshm\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.090511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.090332 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068d660a-401c-4b0a-9717-3a162259439a-tls-certs\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.109639 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.108505 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9tl\" (UniqueName: \"kubernetes.io/projected/068d660a-401c-4b0a-9717-3a162259439a-kube-api-access-ld9tl\") pod \"distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.203065 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.202968 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:42.524149 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.524118 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7"] Oct 29 21:57:42.525251 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:57:42.525222 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod068d660a_401c_4b0a_9717_3a162259439a.slice/crio-752870eaa5a458f758636b260f24f52aa46399407a6bce3335c89fa06e8e87bf WatchSource:0}: Error finding container 752870eaa5a458f758636b260f24f52aa46399407a6bce3335c89fa06e8e87bf: Status 404 returned error can't find the container with id 752870eaa5a458f758636b260f24f52aa46399407a6bce3335c89fa06e8e87bf Oct 29 21:57:42.947499 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:42.947465 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:43.426801 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:43.426761 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" event={"ID":"068d660a-401c-4b0a-9717-3a162259439a","Type":"ContainerStarted","Data":"9e4771a9e36082899bb37ab1561374b28d1a1ce57061cc8bc7315bc10badbb79"} Oct 29 21:57:43.427049 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:43.426811 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" event={"ID":"068d660a-401c-4b0a-9717-3a162259439a","Type":"ContainerStarted","Data":"752870eaa5a458f758636b260f24f52aa46399407a6bce3335c89fa06e8e87bf"} Oct 29 21:57:43.471761 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:43.471703 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podStartSLOduration=2.471687492 podStartE2EDuration="2.471687492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:43.468602641 +0000 UTC m=+928.943191059" watchObservedRunningTime="2026-10-29 21:57:43.471687492 +0000 UTC m=+928.946275911" Oct 29 21:57:44.185583 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:44.185541 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:44.203235 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:44.203184 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:44.756276 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:44.755948 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:45.527597 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:57:45.527552 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:57:46.445614 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.445572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:46.870465 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.870430 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw"] Oct 29 21:57:46.875104 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.875082 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:46.896582 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.896554 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw"] Oct 29 21:57:46.946248 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.946212 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:46.946429 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.946253 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:46.946501 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.946431 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2s25\" (UniqueName: \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-kube-api-access-b2s25\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:46.946501 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.946478 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:46.946617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.946565 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:46.946617 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.946589 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff926c55-e2dc-49b5-9e20-ca7805198914-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:46.946726 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.946706 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:46.946782 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.946762 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:46.946837 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:46.946782 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.048176 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.048138 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.048381 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.048191 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.048381 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.048280 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2s25\" (UniqueName: \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-kube-api-access-b2s25\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.048381 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.048313 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.048381 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.048359 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.048608 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.048386 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff926c55-e2dc-49b5-9e20-ca7805198914-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.048608 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.048484 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.048608 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.048515 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.048608 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.048541 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.049150 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.048834 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.049150 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.049103 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.049333 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.049183 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.049333 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.049269 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff926c55-e2dc-49b5-9e20-ca7805198914-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.049451 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.049383 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.050764 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.050743 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.051018 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.050992 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.073121 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.073099 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.073266 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.073246 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2s25\" (UniqueName: \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-kube-api-access-b2s25\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.189192 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.189114 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:47.516016 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:57:47.515964 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff926c55_e2dc_49b5_9e20_ca7805198914.slice/crio-ec7f1f899addd9d0d4f9ac0151b559ebbeff9238243b948203f8ca5eca666e0d WatchSource:0}: Error finding container ec7f1f899addd9d0d4f9ac0151b559ebbeff9238243b948203f8ca5eca666e0d: Status 404 returned error can't find the container with id ec7f1f899addd9d0d4f9ac0151b559ebbeff9238243b948203f8ca5eca666e0d Oct 29 21:57:47.516228 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.516200 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw"] Oct 29 21:57:47.518612 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.518584 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:57:47.518690 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.518642 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:57:47.518690 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:47.518679 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:57:48.165508 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.165469 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv"] Oct 29 21:57:48.169793 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.169765 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.185670 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.185639 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-82db63-openshift-default-dockercfg-7p225\"" Oct 29 21:57:48.201891 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.201858 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv"] Oct 29 21:57:48.232969 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.232068 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:48.282928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.271571 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.283325 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.283295 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.287206 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.287177 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.287463 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.287444 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blqm6\" (UniqueName: \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-kube-api-access-blqm6\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.287766 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.287750 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d9b425-0133-4d70-a947-7926b68bc0e1-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.294136 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.294109 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.294328 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.294312 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.294467 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.294452 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.294609 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.294595 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.396307 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396269 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqm6\" (UniqueName: \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-kube-api-access-blqm6\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.396501 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396378 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d9b425-0133-4d70-a947-7926b68bc0e1-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.396501 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396430 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.396501 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396445 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.396501 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396476 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.396684 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396511 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.396684 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396546 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.396684 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396582 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.396684 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396625 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.397015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396970 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.397152 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.396986 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.397152 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.397060 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.397301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.397169 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.397615 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.397592 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d9b425-0133-4d70-a947-7926b68bc0e1-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.399041 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.399016 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.399205 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.399185 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.436331 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.427404 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.447106 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.438607 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqm6\" (UniqueName: \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-kube-api-access-blqm6\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.473950 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.473897 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" event={"ID":"ff926c55-e2dc-49b5-9e20-ca7805198914","Type":"ContainerStarted","Data":"4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598"} Oct 29 21:57:48.474133 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.473979 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" event={"ID":"ff926c55-e2dc-49b5-9e20-ca7805198914","Type":"ContainerStarted","Data":"ec7f1f899addd9d0d4f9ac0151b559ebbeff9238243b948203f8ca5eca666e0d"} Oct 29 21:57:48.484580 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.484554 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:48.522930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.521380 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podStartSLOduration=2.521359671 podStartE2EDuration="2.521359671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:48.519358597 +0000 UTC m=+933.993947017" watchObservedRunningTime="2026-10-29 21:57:48.521359671 +0000 UTC m=+933.995948173" Oct 29 21:57:48.854679 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.854636 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv"] Oct 29 21:57:48.864405 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:57:48.864374 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d9b425_0133_4d70_a947_7926b68bc0e1.slice/crio-dd8a679a4d5fd7f20a971fcda0557b10627ca919fed797e5a30658a35fab614e WatchSource:0}: Error finding container dd8a679a4d5fd7f20a971fcda0557b10627ca919fed797e5a30658a35fab614e: Status 404 returned error can't find the container with id dd8a679a4d5fd7f20a971fcda0557b10627ca919fed797e5a30658a35fab614e Oct 29 21:57:48.866984 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.866949 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:57:48.867104 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.867039 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:57:48.867104 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:48.867087 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:57:49.190006 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.189917 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:49.191107 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.191085 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:57:49.191183 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.191142 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 21:57:49.482473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.482394 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" event={"ID":"a4d9b425-0133-4d70-a947-7926b68bc0e1","Type":"ContainerStarted","Data":"79809d795fc4dbe78db8e808d66b7fe263ca0f096d8d349a26b8006598b8a858"} Oct 29 21:57:49.482473 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.482459 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" event={"ID":"a4d9b425-0133-4d70-a947-7926b68bc0e1","Type":"ContainerStarted","Data":"dd8a679a4d5fd7f20a971fcda0557b10627ca919fed797e5a30658a35fab614e"} Oct 29 21:57:49.484842 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.484823 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:57:49.486337 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.486315 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:57:49.486429 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.486368 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 21:57:49.552978 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.542149 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp"] Oct 29 21:57:49.557592 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.557564 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.563035 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.562981 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podStartSLOduration=1.562964925 podStartE2EDuration="1.562964925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:49.544421134 +0000 UTC m=+935.019009565" watchObservedRunningTime="2026-10-29 21:57:49.562964925 +0000 UTC m=+935.037553348" Oct 29 21:57:49.568048 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.568000 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp"] Oct 29 21:57:49.573984 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.573949 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-cea94a-openshift-default-dockercfg-56p6l\"" Oct 29 21:57:49.612228 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.612194 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.612422 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.612330 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6911000-63f9-412a-a097-ed410b7ce727-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.612519 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.612475 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.612600 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.612571 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nczhk\" (UniqueName: \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-kube-api-access-nczhk\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.612739 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.612613 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6911000-63f9-412a-a097-ed410b7ce727-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.612739 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.612694 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.612858 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.612767 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.612858 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.612825 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.613020 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.613002 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.718975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.714054 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.718975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.714142 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nczhk\" (UniqueName: \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-kube-api-access-nczhk\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.718975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.714174 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6911000-63f9-412a-a097-ed410b7ce727-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.718975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.714254 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.718975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.714293 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.718975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.714346 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.718975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.714412 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.718975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.714511 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.718975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.714607 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6911000-63f9-412a-a097-ed410b7ce727-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.718975 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.716257 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.742682 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.742605 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6911000-63f9-412a-a097-ed410b7ce727-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.755944 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.745891 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.755944 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.746587 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6911000-63f9-412a-a097-ed410b7ce727-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.759936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.759728 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.760094 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.760033 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.760297 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.760257 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.789930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.788981 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nczhk\" (UniqueName: \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-kube-api-access-nczhk\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.789930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.789187 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:49.883929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:49.883169 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:50.120867 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:50.108330 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:50.208252 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:50.208213 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:57:50.208771 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:50.208292 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 21:57:50.314214 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:50.300731 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp"] Oct 29 21:57:50.314214 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:50.312391 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:57:50.314214 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:50.312862 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:57:50.314214 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:50.312891 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:57:50.487438 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:50.487377 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:57:50.487629 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:50.487452 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 21:57:50.507618 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:50.507575 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" event={"ID":"a6911000-63f9-412a-a097-ed410b7ce727","Type":"ContainerStarted","Data":"87d27279f3aa8cffa46291a4e40e108acb8d92f231e06be0d80d7aab1ecb5d3d"} Oct 29 21:57:51.209399 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:51.209356 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:57:51.209889 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:51.209434 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 21:57:51.485539 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:51.485439 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:57:51.485799 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:51.485768 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 21:57:51.512471 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:51.512435 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" event={"ID":"a6911000-63f9-412a-a097-ed410b7ce727","Type":"ContainerStarted","Data":"a9fb0b8801753859749f2a764bfef6a36e91d388c3bc2f92983331e5a2adb456"} Oct 29 21:57:51.593059 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:51.592986 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podStartSLOduration=2.5929614130000003 podStartE2EDuration="2.592961413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:51.568613444 +0000 UTC m=+937.043201863" watchObservedRunningTime="2026-10-29 21:57:51.592961413 +0000 UTC m=+937.067549831" Oct 29 21:57:51.888501 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:51.888469 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:57:51.888845 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:51.888826 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:57:51.888995 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:51.888972 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 21:57:52.193369 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:52.193277 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:57:52.193654 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:52.193629 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 21:57:52.209593 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:52.209558 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:52.317735 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:52.317704 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:52.494091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:52.492283 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:57:52.494091 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:52.492358 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 21:57:52.580930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:52.564496 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:52.912933 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:52.892427 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:52.918573 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:52.918095 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:57:52.918573 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:52.918164 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 21:57:53.168819 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:53.168724 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:57:53.194315 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:53.194275 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:57:53.194563 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:53.194533 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 21:57:53.223285 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:53.223240 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:53.383929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:53.382939 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 29 21:57:53.510985 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:53.510007 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:57:53.510985 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:53.510108 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 21:57:53.524930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:53.517380 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:53.898419 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:53.898385 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:57:53.898747 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:53.898715 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 21:57:54.126167 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.126120 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:54.222735 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.222636 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:57:54.223682 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.223646 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 21:57:54.488541 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.488436 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:57:54.488829 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.488782 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 21:57:54.511619 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.511538 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 21:57:54.549286 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.548010 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:54.662226 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.661264 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 29 21:57:54.783981 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.767342 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:54.783981 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.767818 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 21:57:54.787790 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.787748 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 21:57:54.825983 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.825928 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 29 21:57:54.914929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.889138 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:54.914929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.911714 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:57:54.914929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:54.911836 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 21:57:55.081002 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.055841 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:55.142495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.142448 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:55.195259 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.188000 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:55.195259 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.192040 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 29 21:57:55.195259 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.192110 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 29 21:57:55.256331 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.256290 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 21:57:55.412154 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.412049 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:55.436510 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.412387 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:55.501028 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.500977 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:57:55.512965 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.512940 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 21:57:55.513177 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.512849 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:55.564509 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.564367 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 21:57:55.576369 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.576339 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:57:55.610753 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.586456 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:55.749992 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.734024 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:55.749992 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.734394 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:57:55.806534 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.782286 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:55.806534 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.782664 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:57:55.837247 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.837196 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:55.837835 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.837799 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 21:57:55.838459 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.838307 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:57:55.920989 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.920080 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:57:55.920989 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.920145 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 21:57:55.921444 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.921412 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:55.921965 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:55.921947 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:57:56.153194 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.153144 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:56.175986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.154051 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 29 21:57:56.175986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.154404 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 21:57:56.175986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.154885 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:57:56.206930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.188348 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:56.238974 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.207579 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:57:56.298869 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.262438 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:56.303374 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.303346 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:56.416020 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.415512 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:56.438334 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.438294 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 21:57:56.439149 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.439123 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:57:56.481054 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.481007 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:56.481829 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.481804 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:57:56.495684 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.495653 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:57:56.495967 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.495896 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 21:57:56.548418 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.524094 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 21:57:56.550491 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.549639 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:57:56.588320 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.588265 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:56.828987 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.800480 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:56.927926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.837979 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:56.927926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.866659 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 21:57:56.927926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.889475 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:57:56.927926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.922741 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 21:57:56.927926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:56.921674 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:57.039314 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.039281 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:57:57.046924 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.041569 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:57:57.046924 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.041623 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:57:57.105071 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.104970 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 29 21:57:57.158595 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.158564 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:57:57.187928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.165213 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:57.211078 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.174641 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:57.211078 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.209467 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:57:57.424982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.301002 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:57:57.424982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.301065 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:57.424982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.239762 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:57.424982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.341353 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:57.577009 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.562161 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:57.611233 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.611198 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 29 21:57:57.646048 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.646001 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 29 21:57:57.652928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.650658 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:57:57.714234 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.714144 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:57.812939 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.717274 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:57:57.927521 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:57.926269 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 21:57:58.008426 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.008325 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:58.051292 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.051261 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:57:58.159119 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.159088 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:58.187773 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.187749 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:58.196465 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.196444 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:57:58.204929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.202182 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:57:58.204929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.202244 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 21:57:58.210220 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.210192 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 21:57:58.298384 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.298286 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:57:58.301920 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.301710 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:57:58.305596 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.305363 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:57:58.308026 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.307842 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:57:58.585682 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.585648 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:57:58.889290 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.889200 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:57:58.889549 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:58.889516 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 21:57:59.078981 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:59.078943 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:57:59.511954 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:59.511887 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:57:59.512165 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:59.511977 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:57:59.886042 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:59.886005 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 29 21:57:59.886634 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:57:59.886601 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 29 21:58:00.516482 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:00.516444 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:58:00.516680 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:00.516509 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:00.517542 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:00.517517 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:58:00.517865 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:00.517845 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:58:00.518703 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:00.518688 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 21:58:00.888539 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:00.888507 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:58:00.889056 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:00.888922 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:58:00.889605 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:00.889588 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 21:58:01.141083 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:01.140971 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:02.831230 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:02.831190 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:04.651417 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:04.651329 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:06.283957 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:06.280829 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:06.852806 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:06.852774 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:08.016332 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:08.016294 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:09.647370 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.647329 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:09.756442 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.756410 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9"] Oct 29 21:58:09.760567 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.760539 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.798137 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.798107 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9"] Oct 29 21:58:09.828573 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.828535 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.828768 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.828596 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.828768 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.828658 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqgvg\" (UniqueName: \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-kube-api-access-lqgvg\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.828768 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.828690 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.828768 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.828743 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.829070 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.828804 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.829070 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.828874 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.829070 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.828900 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8b8b41-f299-4b37-83f5-82bec45f7517-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.829070 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.829033 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.930198 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.930111 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.930198 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.930179 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.930427 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.930206 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgvg\" (UniqueName: \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-kube-api-access-lqgvg\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.930427 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.930231 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.930427 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.930278 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.930427 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.930311 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.930427 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.930391 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.930427 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.930418 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8b8b41-f299-4b37-83f5-82bec45f7517-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.930727 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.930511 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.930727 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.930625 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.931495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.931192 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.931495 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.931211 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.931831 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.931808 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8b8b41-f299-4b37-83f5-82bec45f7517-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.932083 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.932065 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.932896 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.932874 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.933250 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.933226 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.954636 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.954611 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:09.955082 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:09.955059 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgvg\" (UniqueName: \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-kube-api-access-lqgvg\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:10.078270 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:10.078225 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:10.405596 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:10.405569 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9"] Oct 29 21:58:10.406821 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:58:10.406791 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8b8b41_f299_4b37_83f5_82bec45f7517.slice/crio-3a1f207e4b365828a5ff24f6d6c4bf05c3ed52239c994cd788fbf9998547fc0e WatchSource:0}: Error finding container 3a1f207e4b365828a5ff24f6d6c4bf05c3ed52239c994cd788fbf9998547fc0e: Status 404 returned error can't find the container with id 3a1f207e4b365828a5ff24f6d6c4bf05c3ed52239c994cd788fbf9998547fc0e Oct 29 21:58:10.409333 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:10.409303 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:58:10.409400 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:10.409367 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:58:10.409400 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:10.409393 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:58:10.899023 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:10.898983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:11.081460 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:11.081418 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:11.264265 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:11.264161 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:11.431193 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:11.431147 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" event={"ID":"7f8b8b41-f299-4b37-83f5-82bec45f7517","Type":"ContainerStarted","Data":"5688e7a4cc481d34b4ea40f8118464cf661b602a53edb21604729a3b1066602e"} Oct 29 21:58:11.431660 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:11.431634 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" event={"ID":"7f8b8b41-f299-4b37-83f5-82bec45f7517","Type":"ContainerStarted","Data":"3a1f207e4b365828a5ff24f6d6c4bf05c3ed52239c994cd788fbf9998547fc0e"} Oct 29 21:58:11.512934 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:11.512547 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" podStartSLOduration=2.5125278939999998 podStartE2EDuration="2.512527894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:11.49938652 +0000 UTC m=+956.973974975" watchObservedRunningTime="2026-10-29 21:58:11.512527894 +0000 UTC m=+956.987116314" Oct 29 21:58:11.544131 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:11.544037 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:12.068953 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:12.068468 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:12.080989 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:12.080088 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:12.081823 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:12.081794 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:58:12.082003 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:12.081841 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 21:58:12.252929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:12.249572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:12.440929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:12.440187 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:13.080397 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:13.080354 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:58:13.081005 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:13.080958 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 21:58:13.404285 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:13.404198 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:14.078983 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:14.078952 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:58:14.079148 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:14.079014 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 21:58:15.079159 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:15.079128 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:58:15.079547 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:15.079184 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 21:58:15.237537 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:15.237503 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:15.563439 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:58:15.563340 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:58:16.079435 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:16.079395 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:58:16.079848 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:16.079461 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 21:58:17.056597 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:17.056562 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:17.079877 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:17.079832 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 21:58:17.080418 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:17.079937 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 21:58:18.083697 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:18.083670 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:18.084177 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:18.084156 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:18.084829 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:18.084807 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 21:58:18.759892 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:18.759856 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:20.443099 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:20.443052 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:22.271002 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:22.270954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:23.889085 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:23.889032 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:23.938928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:23.929018 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 21:58:23.938928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:23.930217 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.183626 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.182996 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.214681 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.213989 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:24.337777 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.337706 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 21:58:24.355033 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.354986 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.533291 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.533194 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 21:58:24.558268 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.558225 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 29 21:58:24.604701 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.604657 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 21:58:24.605038 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.604657 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.709046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.708997 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.735934 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.727222 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.759274 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.751033 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:24.803120 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.803011 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.895058 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.894996 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.897033 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.897007 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:58:24.930825 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.929312 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.930825 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.929923 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:58:24.972511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.971052 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:24.972511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:24.971633 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:58:25.161062 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.160995 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.216929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.216084 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.216929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.216520 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 29 21:58:25.224927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.218124 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:58:25.224927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.218185 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:58:25.399159 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.382976 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 21:58:25.399159 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.383703 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:58:25.562342 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.562213 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.600467 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.600427 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.692500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.692466 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:58:25.696129 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.696098 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:58:25.740930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.722557 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 29 21:58:25.740930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.722975 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 21:58:25.740930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.723074 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:58:25.796958 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.773605 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:25.797846 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.797801 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 21:58:25.800926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.798428 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:58:25.805260 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.805237 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:58:25.956020 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.934510 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.364781 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.364737 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 21:58:26.373037 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:25.971854 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.379818 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.087692 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:58:26.391925 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.221063 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:26.392805 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.279966 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.414891 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.414860 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:58:26.470695 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.470657 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:58:26.471103 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.471002 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.506927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.482295 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:58:26.533449 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.533141 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.569137 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.569070 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.767027 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.751190 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.816517 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.815534 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:26.874849 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.874819 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:58:26.875179 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.875163 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:58:26.878620 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.877659 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:58:26.987927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:26.987541 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:27.088167 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:27.088129 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:58:27.090841 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:27.090813 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:58:27.132826 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:27.132798 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:58:27.176521 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:27.176484 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:27.237633 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:27.237566 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:58:27.237978 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:27.237950 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:58:27.271523 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:27.271495 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:58:27.419118 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:27.418992 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:58:28.732212 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:28.732184 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:30.373060 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:30.373011 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:31.375559 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:31.375517 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:32.203631 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:32.203594 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:33.952453 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:33.952414 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:35.672007 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:35.671970 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:36.362303 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:36.362268 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:37.342850 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:37.342812 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:38.705257 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.705224 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl"] Oct 29 21:58:38.709668 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.709646 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.726372 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.726351 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f3b067-openshift-default-dockercfg-t4jcs\"" Oct 29 21:58:38.741051 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.740985 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl"] Oct 29 21:58:38.785447 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.785412 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.785447 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.785449 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.785647 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.785474 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.785647 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.785620 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.785716 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.785659 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.785805 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.785774 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.785876 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.785859 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klwvz\" (UniqueName: \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-kube-api-access-klwvz\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.785960 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.785944 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.786006 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.785975 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca65c2f-2898-48e2-ac26-727c7672ff6d-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.887818 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.887782 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888018 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.887832 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888018 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.887869 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888018 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.887950 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888018 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.888001 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888248 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.888059 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888248 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.888112 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klwvz\" (UniqueName: \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-kube-api-access-klwvz\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888248 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.888154 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888248 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.888186 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca65c2f-2898-48e2-ac26-727c7672ff6d-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.888365 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888683 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.888654 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888683 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.888665 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.888839 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.888816 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.889286 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.889259 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca65c2f-2898-48e2-ac26-727c7672ff6d-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.891156 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.891127 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.891380 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.891359 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.913674 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.913643 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klwvz\" (UniqueName: \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-kube-api-access-klwvz\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:38.914186 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:38.914165 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:39.024381 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:39.024300 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:39.047539 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:39.047459 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:39.369768 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:58:39.369727 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca65c2f_2898_48e2_ac26_727c7672ff6d.slice/crio-ea032c51ccc878083c53a86623004be6862611f83d3b80b64192c878b1a0dbbe WatchSource:0}: Error finding container ea032c51ccc878083c53a86623004be6862611f83d3b80b64192c878b1a0dbbe: Status 404 returned error can't find the container with id ea032c51ccc878083c53a86623004be6862611f83d3b80b64192c878b1a0dbbe Oct 29 21:58:39.374649 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:39.374599 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:58:39.374777 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:39.374743 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:58:39.374853 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:39.374829 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:58:39.382846 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:39.382811 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl"] Oct 29 21:58:40.367941 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:40.366842 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" event={"ID":"cca65c2f-2898-48e2-ac26-727c7672ff6d","Type":"ContainerStarted","Data":"6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934"} Oct 29 21:58:40.367941 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:40.366930 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" event={"ID":"cca65c2f-2898-48e2-ac26-727c7672ff6d","Type":"ContainerStarted","Data":"ea032c51ccc878083c53a86623004be6862611f83d3b80b64192c878b1a0dbbe"} Oct 29 21:58:40.457926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:40.457478 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" podStartSLOduration=2.4574595009999998 podStartE2EDuration="2.457459501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:40.452934768 +0000 UTC m=+985.927523185" watchObservedRunningTime="2026-10-29 21:58:40.457459501 +0000 UTC m=+985.932047922" Oct 29 21:58:40.873498 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:40.873215 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:41.025077 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:41.025047 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:41.043928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:41.036047 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 21:58:41.043928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:41.036134 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 21:58:42.039297 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:42.039260 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 21:58:42.039882 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:42.039848 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 21:58:42.750409 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:42.749453 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:43.028988 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:43.025194 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 21:58:43.028988 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:43.025254 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 21:58:44.025244 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:44.025205 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 21:58:44.025823 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:44.025789 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 21:58:44.664932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:44.653404 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:45.043984 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:45.041302 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 21:58:45.043984 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:45.041364 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 21:58:45.529865 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:58:45.529813 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:58:46.025120 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:46.025085 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 21:58:46.025330 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:46.025144 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 21:58:46.355643 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:46.355602 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:47.015731 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.015696 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 21:58:47.033348 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.033320 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:47.033874 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.033843 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:47.034578 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.034554 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 21:58:47.051519 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.051491 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9"] Oct 29 21:58:47.056677 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.056657 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.096006 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.095969 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9"] Oct 29 21:58:47.125803 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.125771 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgt5k\" (UniqueName: \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-kube-api-access-mgt5k\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.126015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.125852 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35194ac9-c066-4437-b1b1-35defc3e122c-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.126015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.125873 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.126015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.125894 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35194ac9-c066-4437-b1b1-35defc3e122c-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.126015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.125932 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.126015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.125959 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.126284 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.126024 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.126284 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.126076 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.126284 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.126137 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.239547 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgt5k\" (UniqueName: \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-kube-api-access-mgt5k\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.239718 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35194ac9-c066-4437-b1b1-35defc3e122c-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.240671 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35194ac9-c066-4437-b1b1-35defc3e122c-istiod-ca-cert\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.240755 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.240806 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35194ac9-c066-4437-b1b1-35defc3e122c-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.240853 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.240884 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.240974 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.241067 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.241142 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.241968 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.241686 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.242789 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.242767 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-certs\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.243186 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.243163 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-data\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.244389 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.244320 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-envoy\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.247139 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.247113 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-credential-socket\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.247263 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.247230 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35194ac9-c066-4437-b1b1-35defc3e122c-istio-podinfo\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.267865 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.267780 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgt5k\" (UniqueName: \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-kube-api-access-mgt5k\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.270853 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.270814 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-istio-token\") pod \"e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.398122 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.398082 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:47.741563 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.741530 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9"] Oct 29 21:58:47.750049 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.750012 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:58:47.750211 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.750100 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:58:47.750211 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:47.750177 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:58:48.174001 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:48.173965 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:48.443979 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:48.441896 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" event={"ID":"35194ac9-c066-4437-b1b1-35defc3e122c","Type":"ContainerStarted","Data":"bd13e8df9c05bd41165795ecbdbdcf2d4c687ad5d840c498d2bd6a4a9838537f"} Oct 29 21:58:48.443979 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:48.441976 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" event={"ID":"35194ac9-c066-4437-b1b1-35defc3e122c","Type":"ContainerStarted","Data":"db04381aa88064add7f60b997753bb7bf122552184477ef2da57f02c21203a99"} Oct 29 21:58:48.511928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:48.508462 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podStartSLOduration=1.5084384229999999 podStartE2EDuration="1.508438423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:48.487563821 +0000 UTC m=+993.962152240" watchObservedRunningTime="2026-10-29 21:58:48.508438423 +0000 UTC m=+993.983026842" Oct 29 21:58:49.409930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:49.409289 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:58:49.421374 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:49.421033 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:58:49.421374 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:49.421106 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 21:58:49.660133 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:49.660019 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:49.661885 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:49.661849 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:49.779107 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:49.779056 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 21:58:49.924027 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:49.923921 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 21:58:50.077942 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.050295 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 21:58:50.152929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.132961 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.215124 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.215010 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.338118 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.338058 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 21:58:50.398072 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.398024 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.400099 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.400069 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:58:50.400320 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.400293 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 21:58:50.411548 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.411494 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.490438 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.490309 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.590055 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.590005 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.669776 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.669732 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 29 21:58:50.699624 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.683067 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:58:50.699624 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.691205 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 29 21:58:50.699624 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.691827 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:58:50.786630 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.786526 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.787698 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.787668 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:58:50.795024 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.794985 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.934941 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.934203 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:50.935142 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:50.935034 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:58:51.326944 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.301440 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 21:58:51.326944 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.301869 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 21:58:51.326944 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.302887 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:58:51.326944 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.303923 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.355932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.330762 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:51.416138 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.353253 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.416821 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.416786 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:58:51.417157 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.353444 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.417500 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.417438 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:58:51.417821 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.417794 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.418770 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.418753 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:58:51.423936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.422731 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.448235 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.353348 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.492061 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.492008 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.552678 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.552643 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:58:51.568302 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.556049 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 21:58:51.568302 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.554451 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:58:51.568302 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.556169 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:58:51.697013 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.682170 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 29 21:58:51.840528 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.840470 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 29 21:58:51.883023 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.867595 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:51.952996 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.947415 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:51.952996 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:51.948341 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.190152 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.190111 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:58:52.322025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.311235 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:58:52.322025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.312096 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 21:58:52.335930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.323438 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.336357 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.336323 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 21:58:52.356973 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.336766 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 21:58:52.357375 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.337880 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:58:52.357556 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.357542 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:58:52.410927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.407293 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:58:52.410927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.407349 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 21:58:52.445925 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.439539 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.445925 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.439918 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:52.445925 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.441300 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 21:58:52.512928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.512028 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 29 21:58:52.512928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.512844 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.518087 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.513201 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:58:52.615226 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.613588 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 21:58:52.698529 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.683251 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.702318 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.701634 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:58:52.887146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.886018 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:52.953526 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:52.953498 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:58:53.007132 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.007103 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:58:53.110926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.073456 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.310222 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.310135 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:58:53.337269 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.337228 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 21:58:53.374927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.373209 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 29 21:58:53.414305 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.414271 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:58:53.416084 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.416052 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 21:58:53.440475 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.440439 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:53.448257 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.442178 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 29 21:58:53.515929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.513625 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.515929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.513999 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 29 21:58:53.662593 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.662021 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:53.686153 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:53.684992 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.176691 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.176647 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 21:58:54.304061 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.304014 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.339225 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.339181 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.400832 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.400798 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:58:54.401136 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.401105 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 21:58:54.487068 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.486975 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.534662 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.534615 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.535089 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.534937 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 21:58:54.535524 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.534975 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.676343 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.676290 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.687938 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.686230 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.714153 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.714070 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 21:58:54.729146 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.729101 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.806999 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.804048 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:54.894359 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:54.894290 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:55.179122 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.179075 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 21:58:55.189045 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.188989 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:55.339929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.339562 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 21:58:55.406242 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.406190 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:58:55.406512 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.406487 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 21:58:55.408395 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.408368 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:55.504783 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.504681 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 29 21:58:55.536375 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.536332 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:55.537292 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.537261 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 21:58:55.584041 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.583995 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:55.606583 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.606546 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:58:55.607295 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.607271 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:58:55.613147 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.613126 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:58:55.700165 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.700134 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:58:55.703691 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.703665 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:58:55.783982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.779053 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 21:58:55.783982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.779716 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:58:55.783982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.780687 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:58:55.812859 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.812731 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:58:55.901450 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.901280 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:58:55.903159 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:55.903137 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:58:56.408092 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:56.408053 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:58:56.408484 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:56.408116 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 21:58:56.558644 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:56.558611 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:56.757977 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:56.755216 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:57.101510 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:57.101474 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:57.402446 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:57.402368 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:58:57.402692 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:57.402662 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 21:58:57.568197 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:57.567388 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:57.794962 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:57.794814 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:58:58.399044 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:58.399006 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 21:58:58.399257 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:58.399056 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 21:58:59.683626 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:58:59.683569 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:00.399497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:00.399457 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:00.399754 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:00.399721 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:00.778705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:00.778613 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:59:00.779387 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:00.779361 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:59:00.781545 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:00.781509 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 21:59:01.069549 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:01.069459 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:01.267860 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:01.267833 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:01.697949 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:01.697890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:02.782015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:02.781961 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:02.972658 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:02.972599 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:03.161456 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:03.161421 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:03.535206 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:03.535109 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:05.375860 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:05.375816 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:06.935914 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:06.935863 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:07.110880 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:07.110845 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:08.725314 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:08.725282 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:10.425351 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:10.425314 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:12.227370 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:12.227325 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:13.852147 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:13.852106 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:14.026310 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:14.026271 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:14.539151 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:14.539114 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:14.740579 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:14.740539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:15.538058 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:59:15.538016 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:59:15.637120 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.637088 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v"] Oct 29 21:59:15.641657 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.641632 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.656508 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.656474 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-maas-api-tls\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.656657 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.656532 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-metrics-certs\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.656730 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.656698 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e708f6-c85d-440a-9713-b18de6189513-openshift-service-ca\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.656846 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.656829 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25pfz\" (UniqueName: \"kubernetes.io/projected/47e708f6-c85d-440a-9713-b18de6189513-kube-api-access-25pfz\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.675191 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.675154 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a116a2-serving-cert\"" Oct 29 21:59:15.681681 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.679802 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v"] Oct 29 21:59:15.758568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.758534 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-maas-api-tls\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.758772 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.758592 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-metrics-certs\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.758772 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.758672 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e708f6-c85d-440a-9713-b18de6189513-openshift-service-ca\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.758772 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:59:15.758699 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-serving-cert: secret "maas-api-e2e-api-a-a116a2-serving-cert" not found Oct 29 21:59:15.758772 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:59:15.758772 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-maas-api-tls podName:47e708f6-c85d-440a-9713-b18de6189513 nodeName:}" failed. No retries permitted until 2026-10-29 21:59:16.258751419 +0000 UTC m=+1021.733339874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-maas-api-tls") pod "maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" (UID: "47e708f6-c85d-440a-9713-b18de6189513") : secret "maas-api-e2e-api-a-a116a2-serving-cert" not found Oct 29 21:59:15.759044 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.758822 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25pfz\" (UniqueName: \"kubernetes.io/projected/47e708f6-c85d-440a-9713-b18de6189513-kube-api-access-25pfz\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.759449 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.759428 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e708f6-c85d-440a-9713-b18de6189513-openshift-service-ca\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.761787 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.761760 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-metrics-certs\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.784343 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.784316 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25pfz\" (UniqueName: \"kubernetes.io/projected/47e708f6-c85d-440a-9713-b18de6189513-kube-api-access-25pfz\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:15.820449 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:15.820371 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:16.058956 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:16.058918 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:16.267831 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:16.267792 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-maas-api-tls\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:16.270279 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:16.270247 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-maas-api-tls\") pod \"maas-api-e2e-api-a-a116a2-7b49694f97-9f27v\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:16.559218 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:16.558806 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:16.944161 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:16.944128 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v"] Oct 29 21:59:16.945237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:16.945214 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 21:59:17.004130 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.004086 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl"] Oct 29 21:59:17.009609 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.009582 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.059041 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.058998 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl"] Oct 29 21:59:17.084114 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.084075 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.084307 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.084194 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.084307 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.084254 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.084307 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.084274 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4757bc87-b9b1-4f5a-944c-491770f8964c-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.084307 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.084296 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.084538 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.084345 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zr6d\" (UniqueName: \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-kube-api-access-2zr6d\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.084538 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.084471 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.084538 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.084508 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.084715 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.084592 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.186215 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.186176 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.186445 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.186417 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.186511 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.186478 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.186562 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.186546 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.186649 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.186638 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.186685 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.186650 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.186716 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.186685 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.186716 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.186708 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4757bc87-b9b1-4f5a-944c-491770f8964c-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.186778 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.186743 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.186830 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.186813 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zr6d\" (UniqueName: \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-kube-api-access-2zr6d\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.187241 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.187221 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.187430 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.187408 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.188005 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.187984 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4757bc87-b9b1-4f5a-944c-491770f8964c-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.188300 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.188278 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.189686 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.189660 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.189791 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.189739 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.211234 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.211165 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.211387 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.211362 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zr6d\" (UniqueName: \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-kube-api-access-2zr6d\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.330167 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.330127 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:17.559834 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.559739 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" event={"ID":"47e708f6-c85d-440a-9713-b18de6189513","Type":"ContainerStarted","Data":"30eb004e083e03789ec34058605d161ba8ac320e7bc55b8ee0965be4015aa0f4"} Oct 29 21:59:17.559834 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.559803 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" event={"ID":"47e708f6-c85d-440a-9713-b18de6189513","Type":"ContainerStarted","Data":"ed6fce51a3971e81cd43fcf72f91de1659cad4d59f1cea34213e8808dd404577"} Oct 29 21:59:17.559834 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.559827 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:17.606008 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.605957 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" podStartSLOduration=2.201542542 podStartE2EDuration="2.605941053s" totalImagesPullingTime="404.398511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:17.602746983 +0000 UTC m=+1023.077335401" watchObservedRunningTime="2026-10-29 21:59:17.605941053 +0000 UTC m=+1023.080529471" Oct 29 21:59:17.679939 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:59:17.679890 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4757bc87_b9b1_4f5a_944c_491770f8964c.slice/crio-e719c94b08f1229d2aaf83016c2067c55f9f9347cb088ec23621cc25842727f1 WatchSource:0}: Error finding container e719c94b08f1229d2aaf83016c2067c55f9f9347cb088ec23621cc25842727f1: Status 404 returned error can't find the container with id e719c94b08f1229d2aaf83016c2067c55f9f9347cb088ec23621cc25842727f1 Oct 29 21:59:17.680134 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.680115 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl"] Oct 29 21:59:17.682437 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.682406 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:17.682525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.682474 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:17.682525 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.682506 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:17.737618 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:17.737582 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:18.189305 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.189262 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7"] Oct 29 21:59:18.193971 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.193945 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.220971 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.220939 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7"] Oct 29 21:59:18.304110 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.304059 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.304299 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.304131 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtllm\" (UniqueName: \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-kube-api-access-rtllm\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.304299 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.304250 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.304299 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.304283 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.304467 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.304317 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.304467 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.304399 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.304576 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.304501 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e971f8a4-abc5-4235-8abb-4f26d225e35b-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.304633 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.304570 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.304688 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.304630 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406059 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406026 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406232 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406063 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406232 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406118 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e971f8a4-abc5-4235-8abb-4f26d225e35b-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406232 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406168 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406397 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406288 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406471 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406456 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406535 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406468 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406535 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406512 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtllm\" (UniqueName: \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-kube-api-access-rtllm\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406639 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406598 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406692 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406634 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406828 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406804 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.406936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406883 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e971f8a4-abc5-4235-8abb-4f26d225e35b-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.407015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.406993 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.407079 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.407046 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.408635 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.408608 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.408772 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.408754 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.433823 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.433779 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtllm\" (UniqueName: \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-kube-api-access-rtllm\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.435504 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.435468 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.514474 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.514395 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:18.570041 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.569996 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" event={"ID":"4757bc87-b9b1-4f5a-944c-491770f8964c","Type":"ContainerStarted","Data":"cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d"} Oct 29 21:59:18.570522 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.570071 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" event={"ID":"4757bc87-b9b1-4f5a-944c-491770f8964c","Type":"ContainerStarted","Data":"e719c94b08f1229d2aaf83016c2067c55f9f9347cb088ec23621cc25842727f1"} Oct 29 21:59:18.626384 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.626044 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" podStartSLOduration=2.626028455 podStartE2EDuration="2.626028455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:18.619243799 +0000 UTC m=+1024.093832209" watchObservedRunningTime="2026-10-29 21:59:18.626028455 +0000 UTC m=+1024.100616873" Oct 29 21:59:18.863357 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:59:18.863258 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode971f8a4_abc5_4235_8abb_4f26d225e35b.slice/crio-3d636190085d1c2eb8eab6b3ace8af2325aa9253bd5b86f68c618300200df8a7 WatchSource:0}: Error finding container 3d636190085d1c2eb8eab6b3ace8af2325aa9253bd5b86f68c618300200df8a7: Status 404 returned error can't find the container with id 3d636190085d1c2eb8eab6b3ace8af2325aa9253bd5b86f68c618300200df8a7 Oct 29 21:59:18.865620 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.865588 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:18.865719 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.865650 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:18.865719 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.865688 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:18.869216 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:18.869191 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7"] Oct 29 21:59:19.279816 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.279701 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:19.337930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.336278 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:19.354930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.349041 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:19.354930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.349105 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 21:59:19.467381 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.467136 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:19.583969 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.583934 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" event={"ID":"e971f8a4-abc5-4235-8abb-4f26d225e35b","Type":"ContainerStarted","Data":"3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245"} Oct 29 21:59:19.584449 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.584002 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" event={"ID":"e971f8a4-abc5-4235-8abb-4f26d225e35b","Type":"ContainerStarted","Data":"3d636190085d1c2eb8eab6b3ace8af2325aa9253bd5b86f68c618300200df8a7"} Oct 29 21:59:19.644255 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.644190 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" podStartSLOduration=1.644168611 podStartE2EDuration="1.644168611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:19.643429624 +0000 UTC m=+1025.118018055" watchObservedRunningTime="2026-10-29 21:59:19.644168611 +0000 UTC m=+1025.118757030" Oct 29 21:59:19.835986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.834538 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:19.871059 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.871004 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d"] Oct 29 21:59:19.880293 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.880250 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:19.892488 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.892448 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d"] Oct 29 21:59:19.969191 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.969141 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:19.969487 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.969461 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:19.969684 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.969659 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:19.969873 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.969858 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a020b72-5cdf-4826-800c-c43eb3708ec2-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:19.970034 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.970010 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqrw\" (UniqueName: \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-kube-api-access-rnqrw\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:19.970165 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.970151 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:19.970297 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.970277 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:19.970448 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.970427 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:19.970598 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:19.970575 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.079288 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.077792 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a020b72-5cdf-4826-800c-c43eb3708ec2-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.079288 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.077882 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqrw\" (UniqueName: \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-kube-api-access-rnqrw\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.106507 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.106425 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.107046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.107006 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.107322 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.107301 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.107647 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.107628 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.107872 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.107855 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.108152 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.108134 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.108367 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.108350 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.108555 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.108539 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.109254 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.109217 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.109773 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.109754 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.110355 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.110324 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.110696 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.107709 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a020b72-5cdf-4826-800c-c43eb3708ec2-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.114944 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.113428 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.114944 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.114768 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.209740 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.209576 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqrw\" (UniqueName: \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-kube-api-access-rnqrw\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.213959 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.213927 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.214142 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.214122 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:20.294834 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.294800 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:20.335930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.334340 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:20.335930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.334400 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 21:59:20.501645 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.501488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:20.573025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.519570 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:20.573025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.558073 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:20.573025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.558133 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 21:59:20.925937 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.921283 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d"] Oct 29 21:59:20.991699 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.991656 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:20.991809 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.991744 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:20.991809 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:20.991784 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:21.343927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:21.341301 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:21.343927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:21.341364 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 21:59:21.516470 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:21.516432 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:21.516652 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:21.516492 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 21:59:21.714083 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:21.714000 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" event={"ID":"9a020b72-5cdf-4826-800c-c43eb3708ec2","Type":"ContainerStarted","Data":"f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3"} Oct 29 21:59:21.714304 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:21.714287 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" event={"ID":"9a020b72-5cdf-4826-800c-c43eb3708ec2","Type":"ContainerStarted","Data":"eab6c79ae7ca57bf8eb90c23a397e13099d32cfcd8c7e1803a78c513a9b0375e"} Oct 29 21:59:21.773929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:21.765614 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podStartSLOduration=2.765595097 podStartE2EDuration="2.765595097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:21.764502781 +0000 UTC m=+1027.239091224" watchObservedRunningTime="2026-10-29 21:59:21.765595097 +0000 UTC m=+1027.240183516" Oct 29 21:59:22.216283 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:22.216231 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:22.218385 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:22.218349 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 21:59:22.218530 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:22.218412 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 21:59:22.322926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:22.317140 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:22.344928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:22.337406 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:22.344928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:22.337480 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 21:59:22.516986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:22.516195 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:22.516986 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:22.516261 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 21:59:23.237931 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:23.222518 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 21:59:23.237931 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:23.222616 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 21:59:23.348929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:23.343831 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:23.348929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:23.343919 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 21:59:23.517452 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:23.517100 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:23.527928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:23.518970 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:23.527928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:23.519039 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 21:59:23.565932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:23.562071 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:23.581958 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:23.578245 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 21:59:23.696461 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:23.691990 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 29 21:59:24.017037 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.014166 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 29 21:59:24.017037 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.016195 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:24.232070 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.232023 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 21:59:24.242927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.240015 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 21:59:24.323985 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.316014 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 21:59:24.351928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.345171 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:24.351928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.345238 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 21:59:24.576985 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.576692 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 29 21:59:24.588929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.577271 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 21:59:24.588929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.577617 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:59:24.588929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.578002 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:24.588929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.578035 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 21:59:24.618928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.618046 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:24.618928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.618242 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:59:24.656928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.655982 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:24.711927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.707986 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 29 21:59:24.713927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.713014 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:24.763934 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.745236 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:24.810611 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.810576 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:59:24.917497 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.904325 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 21:59:24.945955 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:24.945897 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:25.165961 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.165028 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:25.192314 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.192208 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:25.227928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.221198 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:59:25.234931 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.229841 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 21:59:25.235208 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.235173 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 21:59:25.320165 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.320116 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:25.321803 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.321777 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:59:25.361348 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.348092 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:25.364869 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.364839 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 21:59:25.380884 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.380837 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 21:59:25.587547 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.587506 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 21:59:25.605581 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.605548 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:25.605827 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.605803 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 21:59:25.623711 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.623673 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 29 21:59:25.962121 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.956163 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:25.962121 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.957675 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:59:26.004925 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:25.997051 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:26.204353 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.198973 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 21:59:26.204353 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.200467 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:59:26.204353 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.203271 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:26.204353 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.203568 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:59:26.246982 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.221311 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 21:59:26.247246 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.247213 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 21:59:26.250988 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.247466 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:59:26.298928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.246175 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:26.352930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.324717 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:26.514246 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.514145 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:26.541182 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.526027 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 29 21:59:26.564600 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.564560 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 29 21:59:26.569076 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.563269 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 29 21:59:26.574925 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.563222 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:26.654935 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.652846 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:26.656488 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.656452 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:26.956926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:26.956446 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.128216 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.128170 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.201042 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.200994 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.204038 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.204000 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.231937 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.231834 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 21:59:27.232187 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.232162 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 21:59:27.248072 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.248030 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.320340 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.320300 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.320929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.320885 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:59:27.346157 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.346127 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:27.348771 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.347601 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.349194 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.348969 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:27.355145 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.355123 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 21:59:27.355725 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.355638 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.592121 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.587159 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 29 21:59:27.592121 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.587957 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:59:27.594510 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.588972 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.615929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.607976 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:59:27.678633 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.670504 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 29 21:59:27.678633 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.670842 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.706056 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.696443 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:27.707871 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.707846 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:59:27.900894 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.899083 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:27.961387 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.961338 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 21:59:27.981887 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:27.978524 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.231483 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.231392 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 21:59:28.256301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.256145 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 21:59:28.256301 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.256208 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 21:59:28.259274 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.257163 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.259274 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.257530 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:59:28.285872 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.267008 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 21:59:28.324040 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.323998 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.324803 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.324779 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 21:59:28.428375 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.406690 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.451568 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.451535 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:59:28.466926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.454538 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.664549 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.640600 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.668892 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.668862 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 21:59:28.677676 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.677627 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 29 21:59:28.697851 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.694212 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 21:59:28.705015 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.704979 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:28.705183 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.705040 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.914705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.914608 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:28.916070 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.915570 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:59:28.926883 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.926863 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 21:59:28.963380 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.963249 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:28.965024 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.964505 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:28.965167 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.965101 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 21:59:28.966863 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.966842 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:59:28.975284 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:28.969663 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 21:59:29.040929 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.012460 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.113932 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.112484 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 21:59:29.151616 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.151570 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 21:59:29.207984 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.207862 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 21:59:29.227324 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.218344 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 21:59:29.227324 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.219357 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 21:59:29.227324 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.219547 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 29 21:59:29.227324 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.219607 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 29 21:59:29.245379 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.245344 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 21:59:29.325083 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.325038 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:29.486192 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.486105 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 21:59:29.487939 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.487917 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 21:59:29.488135 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.488123 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 21:59:29.508849 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:29.508208 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 21:59:30.789058 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:30.789014 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:31.298029 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:31.297988 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:31.298319 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:31.298285 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:32.348058 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:32.348000 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 21:59:32.348589 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:32.348082 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:32.610208 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:32.610127 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:32.732462 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:32.732428 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:32.734010 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:32.733988 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:32.741972 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:32.741943 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 21:59:32.875025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:32.860890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:33.370973 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.367065 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94"] Oct 29 21:59:33.391524 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.391491 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:33.404717 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.404170 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94"] Oct 29 21:59:33.404717 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.404282 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.434952 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.433521 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 21:59:33.435237 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.435219 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 21:59:33.443228 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.435380 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 21:59:33.443425 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.435485 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wt9dr\"" Oct 29 21:59:33.540726 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.540690 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fa5b64-54c9-4cb9-aa39-771246749378-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.541122 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.541101 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdfz\" (UniqueName: \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-kube-api-access-lkdfz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.541294 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.541268 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.541490 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.541474 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.541688 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.541673 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.541801 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.541788 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.541981 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.541964 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.581560 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.581509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:33.643059 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.642973 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdfz\" (UniqueName: \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-kube-api-access-lkdfz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.643346 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.643322 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.643566 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.643548 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.643807 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.643788 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.643949 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.643933 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.644120 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.644101 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.644261 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.644246 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fa5b64-54c9-4cb9-aa39-771246749378-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.655479 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.655452 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.659921 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.659680 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.661661 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.661618 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.664899 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.662429 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.668332 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.668300 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.670136 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.670112 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdfz\" (UniqueName: \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-kube-api-access-lkdfz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.670455 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.670431 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fa5b64-54c9-4cb9-aa39-771246749378-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8rs94\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:33.752178 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:33.752140 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:34.111140 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:34.111108 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94"] Oct 29 21:59:34.112641 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:59:34.112596 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fa5b64_54c9_4cb9_aa39_771246749378.slice/crio-e434a9ecdd70d659e8e88b88062175239d778a8daa9356a5c422d9d3c7e5fd2d WatchSource:0}: Error finding container e434a9ecdd70d659e8e88b88062175239d778a8daa9356a5c422d9d3c7e5fd2d: Status 404 returned error can't find the container with id e434a9ecdd70d659e8e88b88062175239d778a8daa9356a5c422d9d3c7e5fd2d Oct 29 21:59:34.368987 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:34.368862 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" event={"ID":"64fa5b64-54c9-4cb9-aa39-771246749378","Type":"ContainerStarted","Data":"e434a9ecdd70d659e8e88b88062175239d778a8daa9356a5c422d9d3c7e5fd2d"} Oct 29 21:59:34.903864 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:34.897657 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:36.842782 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:36.840155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:37.262988 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:37.260834 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:37.322467 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:37.322432 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:37.322578 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:37.322501 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:37.433814 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:37.433777 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:38.410046 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:38.410003 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" event={"ID":"64fa5b64-54c9-4cb9-aa39-771246749378","Type":"ContainerStarted","Data":"df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de"} Oct 29 21:59:38.411415 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:38.411256 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:38.417949 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:38.415069 4952 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8rs94 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 21:59:38.417949 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:38.415154 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" podUID="64fa5b64-54c9-4cb9-aa39-771246749378" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 21:59:38.738717 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:38.738611 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:38.914255 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:38.914220 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:39.417179 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:39.417147 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 21:59:39.491096 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:39.491021 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" podStartSLOduration=3.284691768 podStartE2EDuration="6.491003071s" totalImagesPullingTime="3.206311303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:38.49558599 +0000 UTC m=+1043.970174400" watchObservedRunningTime="2026-10-29 21:59:39.491003071 +0000 UTC m=+1044.965591486" Oct 29 21:59:40.710526 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:40.710473 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:40.886118 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:40.886085 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:41.081361 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:41.081330 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:41.292234 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:41.292201 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:41.723215 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:41.722836 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:42.379194 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:42.379162 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:42.648509 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:42.648422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:42.867955 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:42.866545 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:43.211561 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:43.211525 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:44.421643 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:44.421608 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:45.527710 ip-10-0-130-134 kubenswrapper[4952]: E1029 21:59:45.527672 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 21:59:46.092466 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:46.092431 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:46.270299 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:46.270214 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:47.789712 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:47.789672 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:47.985804 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:47.985771 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:48.167520 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:48.167481 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:49.841757 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:49.841713 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:49.976400 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:49.976367 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9"] Oct 29 21:59:49.987312 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:49.987284 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.011950 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.011920 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9"] Oct 29 21:59:50.105147 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.105067 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.105147 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.105104 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.105147 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.105121 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.105364 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.105288 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.105401 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.105363 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.105434 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.105408 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.105553 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.105536 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5f7m\" (UniqueName: \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-kube-api-access-k5f7m\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.105641 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.105618 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.105676 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.105651 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d970934-54ed-4252-b585-1a0d34aeeaa9-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.206852 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.206818 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.207068 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.206860 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d970934-54ed-4252-b585-1a0d34aeeaa9-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.207068 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.206896 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.207068 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.206956 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.207241 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.207058 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.207241 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.207214 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.207335 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.207280 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.207335 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.207315 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-credential-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.207428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.207350 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.207548 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.207448 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5f7m\" (UniqueName: \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-kube-api-access-k5f7m\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.207637 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.207589 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-data\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.208005 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.207691 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d970934-54ed-4252-b585-1a0d34aeeaa9-istiod-ca-cert\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.208005 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.207953 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-socket\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.208005 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.207980 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-certs\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.209519 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.209496 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-envoy\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.209830 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.209809 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-podinfo\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.232641 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.232612 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5f7m\" (UniqueName: \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-kube-api-access-k5f7m\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.233079 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.233062 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-token\") pod \"e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.315555 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.315527 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:50.652739 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.652709 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9"] Oct 29 21:59:50.654031 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:59:50.653999 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d970934_54ed_4252_b585_1a0d34aeeaa9.slice/crio-33254a9d6be0e480a6339ae3e41f761aa662c39a96fa2237830c1c62503e460c WatchSource:0}: Error finding container 33254a9d6be0e480a6339ae3e41f761aa662c39a96fa2237830c1c62503e460c: Status 404 returned error can't find the container with id 33254a9d6be0e480a6339ae3e41f761aa662c39a96fa2237830c1c62503e460c Oct 29 21:59:50.656966 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.656891 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:50.657096 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.656996 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:50.657096 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:50.657027 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:51.514443 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:51.514405 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" event={"ID":"8d970934-54ed-4252-b585-1a0d34aeeaa9","Type":"ContainerStarted","Data":"895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368"} Oct 29 21:59:51.514878 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:51.514490 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" event={"ID":"8d970934-54ed-4252-b585-1a0d34aeeaa9","Type":"ContainerStarted","Data":"33254a9d6be0e480a6339ae3e41f761aa662c39a96fa2237830c1c62503e460c"} Oct 29 21:59:51.590299 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:51.590243 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" podStartSLOduration=2.590228764 podStartE2EDuration="2.590228764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:51.575282731 +0000 UTC m=+1057.049871148" watchObservedRunningTime="2026-10-29 21:59:51.590228764 +0000 UTC m=+1057.064817181" Oct 29 21:59:51.646970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:51.641757 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:52.316518 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:52.316478 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:52.317813 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:52.317788 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:59:52.317947 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:52.317862 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 21:59:53.316486 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:53.316437 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:59:53.316900 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:53.316505 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 21:59:53.391667 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:53.391600 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:54.033970 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.033929 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg"] Oct 29 21:59:54.049972 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.049944 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.066930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.062660 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj"] Oct 29 21:59:54.081705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.080950 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.081705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.081046 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.081705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.081112 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.081705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.081146 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjt8b\" (UniqueName: \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-kube-api-access-pjt8b\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.081705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.081175 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.081705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.081201 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.081705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.081371 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.081705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.081498 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.081705 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.081577 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ca98e1-3645-45e0-a041-66af7c97da96-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.091023 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.089227 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg"] Oct 29 21:59:54.091023 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.089330 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.099126 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.099055 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj"] Oct 29 21:59:54.182597 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.182559 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.182796 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.182612 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.182796 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.182675 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkxdz\" (UniqueName: \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-kube-api-access-rkxdz\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.182796 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.182759 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.182796 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.182791 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.183382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.182813 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjt8b\" (UniqueName: \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-kube-api-access-pjt8b\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.183382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.182830 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.183382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.182888 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.183382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.182947 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.183382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183073 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.183382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183143 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.183382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183151 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.183382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183218 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.183382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183232 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.183382 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183366 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.183839 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183401 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.183839 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183435 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.183839 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183451 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.183839 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183486 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ca98e1-3645-45e0-a041-66af7c97da96-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.183839 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183507 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cde5d96-3692-421c-81b2-44ad13d35d1e-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.183839 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.183785 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.184071 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.184023 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.184216 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.184198 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ca98e1-3645-45e0-a041-66af7c97da96-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.185424 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.185400 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.185533 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.185517 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.207225 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.207202 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.207428 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.207410 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjt8b\" (UniqueName: \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-kube-api-access-pjt8b\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.284977 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.284877 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.285127 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.284979 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkxdz\" (UniqueName: \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-kube-api-access-rkxdz\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.285127 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.285063 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.285127 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.285102 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.285242 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.285192 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.285242 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.285231 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.285322 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.285298 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.285418 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.285357 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.285418 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.285396 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cde5d96-3692-421c-81b2-44ad13d35d1e-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.286025 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.285995 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.286212 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.286043 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.286309 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.286059 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.286391 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.286257 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cde5d96-3692-421c-81b2-44ad13d35d1e-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.286483 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.286278 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.289073 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.289048 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.289313 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.289293 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.312530 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.312455 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.312899 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.312879 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkxdz\" (UniqueName: \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-kube-api-access-rkxdz\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.316345 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.316324 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:59:54.316420 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.316378 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 21:59:54.399284 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.399248 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:54.410416 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.410394 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:54.796333 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.796305 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg"] Oct 29 21:59:54.796615 ip-10-0-130-134 kubenswrapper[4952]: W1029 21:59:54.796586 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ca98e1_3645_45e0_a041_66af7c97da96.slice/crio-667188ffebf416801788c13a20bbdceffba55675972c3418d1b0293042614b4b WatchSource:0}: Error finding container 667188ffebf416801788c13a20bbdceffba55675972c3418d1b0293042614b4b: Status 404 returned error can't find the container with id 667188ffebf416801788c13a20bbdceffba55675972c3418d1b0293042614b4b Oct 29 21:59:54.799321 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.799287 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:54.799422 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.799360 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:54.799422 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.799403 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:54.799795 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.799769 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj"] Oct 29 21:59:54.816206 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.816165 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:54.816305 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.816254 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:54.816305 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:54.816296 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 21:59:55.319575 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:55.316258 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:59:55.319575 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:55.316314 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 21:59:55.376930 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:55.371484 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:55.597288 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:55.597245 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:55.609541 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:55.609501 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" event={"ID":"c0ca98e1-3645-45e0-a041-66af7c97da96","Type":"ContainerStarted","Data":"ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575"} Oct 29 21:59:55.609860 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:55.609831 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" event={"ID":"c0ca98e1-3645-45e0-a041-66af7c97da96","Type":"ContainerStarted","Data":"667188ffebf416801788c13a20bbdceffba55675972c3418d1b0293042614b4b"} Oct 29 21:59:55.611870 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:55.611846 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" event={"ID":"7cde5d96-3692-421c-81b2-44ad13d35d1e","Type":"ContainerStarted","Data":"3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269"} Oct 29 21:59:55.611947 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:55.611889 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" event={"ID":"7cde5d96-3692-421c-81b2-44ad13d35d1e","Type":"ContainerStarted","Data":"f2adb4d2a4cbc3e17624bfd6e33696636ecf9bf0aca0630d5f541bbe09c16a78"} Oct 29 21:59:55.679924 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:55.678965 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" podStartSLOduration=2.678944751 podStartE2EDuration="2.678944751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:55.677540612 +0000 UTC m=+1061.152129041" watchObservedRunningTime="2026-10-29 21:59:55.678944751 +0000 UTC m=+1061.153533170" Oct 29 21:59:55.771924 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:55.771421 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podStartSLOduration=1.771400721 podStartE2EDuration="1.771400721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:55.765751052 +0000 UTC m=+1061.240339471" watchObservedRunningTime="2026-10-29 21:59:55.771400721 +0000 UTC m=+1061.245989151" Oct 29 21:59:56.317031 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:56.316310 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:59:56.317031 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:56.316376 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 21:59:56.408931 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:56.403751 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 21:59:56.408931 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:56.403883 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 21:59:56.408931 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:56.403950 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 21:59:56.416926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:56.414538 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 21:59:56.416926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:56.414700 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 21:59:56.416926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:56.414747 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 21:59:57.209927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.209300 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:57.294928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.294047 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:57.338926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.338622 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:59:57.338926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.338712 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 21:59:57.361927 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.339205 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:57.383936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.376991 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:57.408936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.407947 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 21:59:57.408936 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.408005 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 21:59:57.417805 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.417689 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 21:59:57.417805 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.417743 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 21:59:57.511430 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.496315 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 21:59:57.971785 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:57.971018 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 21:59:58.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:58.201561 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 21:59:58.318994 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:58.316676 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 21:59:58.318994 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:58.318175 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 21:59:58.406143 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:58.404262 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 21:59:58.406143 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:58.404324 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 21:59:58.415079 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:58.415039 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 21:59:58.415269 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:58.415097 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 21:59:59.485928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:59.418061 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 21:59:59.485928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:59.418173 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 21:59:59.485928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:59.419689 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 21:59:59.485928 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:59.419789 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 21:59:59.787990 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:59.735746 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:59.787990 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:59.759194 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 21:59:59.787990 ip-10-0-130-134 kubenswrapper[4952]: I1029 21:59:59.762717 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 22:00:00.193928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.187103 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4"] Oct 29 22:00:00.210981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.210178 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc"] Oct 29 22:00:00.216416 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.216389 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" Oct 29 22:00:00.307164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.306527 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:00:00.307873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.307843 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4\" (UID: \"e09dc521-2f96-4a1b-aa00-2a28419fdfd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" Oct 29 22:00:00.308043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.307964 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2tsx\" (UniqueName: \"kubernetes.io/projected/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-kube-api-access-k2tsx\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4\" (UID: \"e09dc521-2f96-4a1b-aa00-2a28419fdfd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" Oct 29 22:00:00.310184 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.310156 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv"] Oct 29 22:00:00.311225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.310282 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" Oct 29 22:00:00.463342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.419839 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:00:00.463342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.419942 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 22:00:00.505700 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.505512 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:00.557929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.543771 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vs8m\" (UniqueName: \"kubernetes.io/projected/a552076f-7163-4206-a7b6-ae05b998d9b6-kube-api-access-8vs8m\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc\" (UID: \"a552076f-7163-4206-a7b6-ae05b998d9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" Oct 29 22:00:00.557929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.543969 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a552076f-7163-4206-a7b6-ae05b998d9b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc\" (UID: \"a552076f-7163-4206-a7b6-ae05b998d9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" Oct 29 22:00:00.557929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.544013 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4\" (UID: \"e09dc521-2f96-4a1b-aa00-2a28419fdfd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" Oct 29 22:00:00.557929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.544112 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2tsx\" (UniqueName: \"kubernetes.io/projected/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-kube-api-access-k2tsx\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4\" (UID: \"e09dc521-2f96-4a1b-aa00-2a28419fdfd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" Oct 29 22:00:00.567933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.563607 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4\" (UID: \"e09dc521-2f96-4a1b-aa00-2a28419fdfd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" Oct 29 22:00:00.567933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.565299 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4"] Oct 29 22:00:00.567933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.565346 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp"] Oct 29 22:00:00.593930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.571424 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" Oct 29 22:00:00.601117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.601087 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:00:00.601324 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.601298 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:00:00.607781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.607753 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2tsx\" (UniqueName: \"kubernetes.io/projected/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-kube-api-access-k2tsx\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4\" (UID: \"e09dc521-2f96-4a1b-aa00-2a28419fdfd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" Oct 29 22:00:00.608527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.608504 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc"] Oct 29 22:00:00.608653 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.608564 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv"] Oct 29 22:00:00.608653 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.608591 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp"] Oct 29 22:00:00.608757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.608713 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" Oct 29 22:00:00.617792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.617768 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" Oct 29 22:00:00.666856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.666817 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vs8m\" (UniqueName: \"kubernetes.io/projected/a552076f-7163-4206-a7b6-ae05b998d9b6-kube-api-access-8vs8m\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc\" (UID: \"a552076f-7163-4206-a7b6-ae05b998d9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" Oct 29 22:00:00.667270 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.667247 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a552076f-7163-4206-a7b6-ae05b998d9b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc\" (UID: \"a552076f-7163-4206-a7b6-ae05b998d9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" Oct 29 22:00:00.684928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.668440 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a552076f-7163-4206-a7b6-ae05b998d9b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc\" (UID: \"a552076f-7163-4206-a7b6-ae05b998d9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" Oct 29 22:00:00.789979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.772099 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsclg\" (UniqueName: \"kubernetes.io/projected/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-kube-api-access-xsclg\") pod \"maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv\" (UID: \"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" Oct 29 22:00:00.789979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.772274 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwkzw\" (UniqueName: \"kubernetes.io/projected/8925c000-d6eb-4bee-beb6-526447d50471-kube-api-access-kwkzw\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp\" (UID: \"8925c000-d6eb-4bee-beb6-526447d50471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" Oct 29 22:00:00.789979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.772336 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8925c000-d6eb-4bee-beb6-526447d50471-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp\" (UID: \"8925c000-d6eb-4bee-beb6-526447d50471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" Oct 29 22:00:00.789979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.772421 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv\" (UID: \"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" Oct 29 22:00:00.815730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.814726 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vs8m\" (UniqueName: \"kubernetes.io/projected/a552076f-7163-4206-a7b6-ae05b998d9b6-kube-api-access-8vs8m\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc\" (UID: \"a552076f-7163-4206-a7b6-ae05b998d9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" Oct 29 22:00:00.881927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.877777 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8925c000-d6eb-4bee-beb6-526447d50471-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp\" (UID: \"8925c000-d6eb-4bee-beb6-526447d50471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" Oct 29 22:00:00.903967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.883084 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8925c000-d6eb-4bee-beb6-526447d50471-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp\" (UID: \"8925c000-d6eb-4bee-beb6-526447d50471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" Oct 29 22:00:00.903967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.900686 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv\" (UID: \"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" Oct 29 22:00:00.903967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.901541 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv\" (UID: \"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" Oct 29 22:00:00.903967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.901884 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsclg\" (UniqueName: \"kubernetes.io/projected/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-kube-api-access-xsclg\") pod \"maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv\" (UID: \"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" Oct 29 22:00:00.903967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.902183 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwkzw\" (UniqueName: \"kubernetes.io/projected/8925c000-d6eb-4bee-beb6-526447d50471-kube-api-access-kwkzw\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp\" (UID: \"8925c000-d6eb-4bee-beb6-526447d50471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" Oct 29 22:00:00.909753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:00.909726 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" Oct 29 22:00:01.015224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.015166 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsclg\" (UniqueName: \"kubernetes.io/projected/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-kube-api-access-xsclg\") pod \"maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv\" (UID: \"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" Oct 29 22:00:01.016696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.016669 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwkzw\" (UniqueName: \"kubernetes.io/projected/8925c000-d6eb-4bee-beb6-526447d50471-kube-api-access-kwkzw\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp\" (UID: \"8925c000-d6eb-4bee-beb6-526447d50471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" Oct 29 22:00:01.107765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.107729 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" Oct 29 22:00:01.208481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.203581 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" Oct 29 22:00:01.208481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.207301 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4"] Oct 29 22:00:01.402068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.402037 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:00:01.402167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.402097 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:00:01.414800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.413130 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:00:01.414800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.413188 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 22:00:01.707606 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.701975 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc"] Oct 29 22:00:01.708262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.708236 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45"] Oct 29 22:00:01.711533 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:00:01.711500 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda552076f_7163_4206_a7b6_ae05b998d9b6.slice/crio-c3e92bba66ada2352281f82d12edfb2ed3e39726deb20cf801cd11daee9ed3dd WatchSource:0}: Error finding container c3e92bba66ada2352281f82d12edfb2ed3e39726deb20cf801cd11daee9ed3dd: Status 404 returned error can't find the container with id c3e92bba66ada2352281f82d12edfb2ed3e39726deb20cf801cd11daee9ed3dd Oct 29 22:00:01.715696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.714696 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.745930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.745526 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-863f1b-openshift-default-dockercfg-5qt7r\"" Oct 29 22:00:01.810709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.803011 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.810709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.803102 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.810709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.803174 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc40683-1682-4064-a856-b7c8dd137a0a-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.810709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.803217 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.810709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.803284 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.810709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.803361 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.810709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.803405 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4264\" (UniqueName: \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-kube-api-access-c4264\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.810709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.803449 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.810709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.803526 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.812300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.812271 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45"] Oct 29 22:00:01.899467 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.898213 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv"] Oct 29 22:00:01.905307 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.905225 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.905425 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.905306 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.909784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.905372 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc40683-1682-4064-a856-b7c8dd137a0a-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.910750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.905856 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-data\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.910750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.906250 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc40683-1682-4064-a856-b7c8dd137a0a-istiod-ca-cert\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.911389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.910989 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.911606 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.911577 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.914944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.913946 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.914944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.914003 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4264\" (UniqueName: \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-kube-api-access-c4264\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.914944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.914043 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.914944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.914124 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.914944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.914522 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-envoy\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.914944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.914615 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-certs\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.914944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.914897 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-credential-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.915332 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.915027 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-socket\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.917738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.917715 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-podinfo\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.980220 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.980119 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4264\" (UniqueName: \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-kube-api-access-c4264\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:01.987230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:01.985737 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-token\") pod \"e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:02.017578 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.017096 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp"] Oct 29 22:00:02.018048 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:00:02.017996 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8925c000_d6eb_4bee_beb6_526447d50471.slice/crio-f98e65580c1bf8405a14c6553492225d4c9cb927ce7e40eb81486d18321a30a6 WatchSource:0}: Error finding container f98e65580c1bf8405a14c6553492225d4c9cb927ce7e40eb81486d18321a30a6: Status 404 returned error can't find the container with id f98e65580c1bf8405a14c6553492225d4c9cb927ce7e40eb81486d18321a30a6 Oct 29 22:00:02.137157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.131746 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:02.198973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.188604 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" event={"ID":"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca","Type":"ContainerStarted","Data":"e746e2f3ed08077258ba8338c85e4be8fdeb9a2ff34b750325e10c18cf00ca8d"} Oct 29 22:00:02.198973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.190248 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" event={"ID":"8925c000-d6eb-4bee-beb6-526447d50471","Type":"ContainerStarted","Data":"f98e65580c1bf8405a14c6553492225d4c9cb927ce7e40eb81486d18321a30a6"} Oct 29 22:00:02.198973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.191769 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" event={"ID":"a552076f-7163-4206-a7b6-ae05b998d9b6","Type":"ContainerStarted","Data":"c3e92bba66ada2352281f82d12edfb2ed3e39726deb20cf801cd11daee9ed3dd"} Oct 29 22:00:02.198973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.193498 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" event={"ID":"e09dc521-2f96-4a1b-aa00-2a28419fdfd2","Type":"ContainerStarted","Data":"dce5f8c53c715fdf83fa9528065f035a455c8e053c96fdf568dcd6b8c52c41a6"} Oct 29 22:00:02.198973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.193546 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" event={"ID":"e09dc521-2f96-4a1b-aa00-2a28419fdfd2","Type":"ContainerStarted","Data":"6eddcc25d151029a3f3fff64c6f3762ec316f3d3436cd9923a814edfd23828f6"} Oct 29 22:00:02.205231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.205198 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:00:02.267189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.267123 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" podStartSLOduration=2.267102217 podStartE2EDuration="2.267102217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:02.256974061 +0000 UTC m=+1067.731562476" watchObservedRunningTime="2026-10-29 22:00:02.267102217 +0000 UTC m=+1067.741690637" Oct 29 22:00:02.413458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.411896 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:00:02.413458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.411986 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 22:00:02.413458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.413071 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:00:02.413458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.413114 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 29 22:00:02.667397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.667301 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:02.843181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.842487 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45"] Oct 29 22:00:02.848934 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:00:02.848882 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc40683_1682_4064_a856_b7c8dd137a0a.slice/crio-c542b508fa463a18896ee24ff5b91451f2924c233f984dbf68b36f0a3633de0c WatchSource:0}: Error finding container c542b508fa463a18896ee24ff5b91451f2924c233f984dbf68b36f0a3633de0c: Status 404 returned error can't find the container with id c542b508fa463a18896ee24ff5b91451f2924c233f984dbf68b36f0a3633de0c Oct 29 22:00:02.852738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.852687 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:00:02.854949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.853036 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:00:02.854949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:02.853092 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:00:03.284407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.283982 4952 generic.go:356] "Generic (PLEG): container finished" podID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerID="7d4cfd8878a10d5a7ae62bcc999571c96d164930b06c585a28be4d1a8cf62bda" exitCode=60 Oct 29 22:00:03.284407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.284203 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" event={"ID":"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca","Type":"ContainerDied","Data":"7d4cfd8878a10d5a7ae62bcc999571c96d164930b06c585a28be4d1a8cf62bda"} Oct 29 22:00:03.302695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.302322 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" event={"ID":"8925c000-d6eb-4bee-beb6-526447d50471","Type":"ContainerStarted","Data":"646fad0cc6a7b33fe428291835e83c8ef8cf8dce027033eaf06a1bc043bff6a7"} Oct 29 22:00:03.306667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.304926 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" event={"ID":"a552076f-7163-4206-a7b6-ae05b998d9b6","Type":"ContainerStarted","Data":"61d0d838c2a486eca7ac9aa749e5f00a0bca9daa656a4acf6bdaf867ecb7eb3a"} Oct 29 22:00:03.308696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.308537 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" event={"ID":"7dc40683-1682-4064-a856-b7c8dd137a0a","Type":"ContainerStarted","Data":"d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7"} Oct 29 22:00:03.308696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.308576 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" event={"ID":"7dc40683-1682-4064-a856-b7c8dd137a0a","Type":"ContainerStarted","Data":"c542b508fa463a18896ee24ff5b91451f2924c233f984dbf68b36f0a3633de0c"} Oct 29 22:00:03.310370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.310143 4952 generic.go:356] "Generic (PLEG): container finished" podID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerID="dce5f8c53c715fdf83fa9528065f035a455c8e053c96fdf568dcd6b8c52c41a6" exitCode=60 Oct 29 22:00:03.310370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.310220 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" event={"ID":"e09dc521-2f96-4a1b-aa00-2a28419fdfd2","Type":"ContainerDied","Data":"dce5f8c53c715fdf83fa9528065f035a455c8e053c96fdf568dcd6b8c52c41a6"} Oct 29 22:00:03.323214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.322481 4952 scope.go:122] "RemoveContainer" containerID="7d4cfd8878a10d5a7ae62bcc999571c96d164930b06c585a28be4d1a8cf62bda" Oct 29 22:00:03.349207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.340793 4952 scope.go:122] "RemoveContainer" containerID="dce5f8c53c715fdf83fa9528065f035a455c8e053c96fdf568dcd6b8c52c41a6" Oct 29 22:00:03.387684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.387561 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" podStartSLOduration=3.387543808 podStartE2EDuration="3.387543808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:03.378587104 +0000 UTC m=+1068.853175545" watchObservedRunningTime="2026-10-29 22:00:03.387543808 +0000 UTC m=+1068.862132226" Oct 29 22:00:03.411463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.411414 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:00:03.411848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.411811 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 22:00:03.610973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.600653 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podStartSLOduration=2.600631522 podStartE2EDuration="2.600631522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:03.494961211 +0000 UTC m=+1068.969549640" watchObservedRunningTime="2026-10-29 22:00:03.600631522 +0000 UTC m=+1069.075219941" Oct 29 22:00:03.610973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:03.601552 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" podStartSLOduration=3.601541941 podStartE2EDuration="3.601541941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:03.599246441 +0000 UTC m=+1069.073834885" watchObservedRunningTime="2026-10-29 22:00:03.601541941 +0000 UTC m=+1069.076130359" Oct 29 22:00:04.084928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.082142 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:04.144976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.141812 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:00:04.144976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.141883 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:00:04.144976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.142178 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:04.439317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.431300 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:00:04.439317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.431374 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:04.609040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.598695 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" event={"ID":"e09dc521-2f96-4a1b-aa00-2a28419fdfd2","Type":"ContainerStarted","Data":"b6057a55ff29f412fd5ee5c41de5ef3ae38d5732b2824cfdda10fcdfd6aeaa8b"} Oct 29 22:00:04.609040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.606538 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" event={"ID":"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca","Type":"ContainerStarted","Data":"97378b9fb52c3d27ecd1d4351fbd1aa38ba48b3c9fec2b3849311e96612bfa65"} Oct 29 22:00:04.669887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.631346 4952 generic.go:356] "Generic (PLEG): container finished" podID="8925c000-d6eb-4bee-beb6-526447d50471" containerID="646fad0cc6a7b33fe428291835e83c8ef8cf8dce027033eaf06a1bc043bff6a7" exitCode=60 Oct 29 22:00:04.669887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.632032 4952 scope.go:122] "RemoveContainer" containerID="646fad0cc6a7b33fe428291835e83c8ef8cf8dce027033eaf06a1bc043bff6a7" Oct 29 22:00:04.669887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.640758 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" event={"ID":"8925c000-d6eb-4bee-beb6-526447d50471","Type":"ContainerDied","Data":"646fad0cc6a7b33fe428291835e83c8ef8cf8dce027033eaf06a1bc043bff6a7"} Oct 29 22:00:04.724771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.710742 4952 generic.go:356] "Generic (PLEG): container finished" podID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerID="61d0d838c2a486eca7ac9aa749e5f00a0bca9daa656a4acf6bdaf867ecb7eb3a" exitCode=60 Oct 29 22:00:04.724771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.712213 4952 scope.go:122] "RemoveContainer" containerID="61d0d838c2a486eca7ac9aa749e5f00a0bca9daa656a4acf6bdaf867ecb7eb3a" Oct 29 22:00:04.907229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.906546 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" event={"ID":"a552076f-7163-4206-a7b6-ae05b998d9b6","Type":"ContainerDied","Data":"61d0d838c2a486eca7ac9aa749e5f00a0bca9daa656a4acf6bdaf867ecb7eb3a"} Oct 29 22:00:04.907229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.906650 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 22:00:04.916070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.907751 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 22:00:04.934302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:04.931183 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 22:00:05.056149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.056055 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.119423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.119377 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.140431 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.140394 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:00:05.140577 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.140458 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:00:05.207805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.207738 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" podStartSLOduration=5.207715768 podStartE2EDuration="5.207715768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:05.18703709 +0000 UTC m=+1070.661625511" watchObservedRunningTime="2026-10-29 22:00:05.207715768 +0000 UTC m=+1070.682304187" Oct 29 22:00:05.239414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.208136 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.404929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.380815 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 22:00:05.518962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.499816 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.535343 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.522314 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.535343 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.523457 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:00:05.535343 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.534053 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:00:05.535343 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.534127 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:05.705000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.697472 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:05.985945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:05.984135 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:06.007169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.007121 4952 generic.go:356] "Generic (PLEG): container finished" podID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerID="b6057a55ff29f412fd5ee5c41de5ef3ae38d5732b2824cfdda10fcdfd6aeaa8b" exitCode=60 Oct 29 22:00:06.007728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.007701 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" event={"ID":"e09dc521-2f96-4a1b-aa00-2a28419fdfd2","Type":"ContainerDied","Data":"b6057a55ff29f412fd5ee5c41de5ef3ae38d5732b2824cfdda10fcdfd6aeaa8b"} Oct 29 22:00:06.008014 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.007997 4952 scope.go:122] "RemoveContainer" containerID="dce5f8c53c715fdf83fa9528065f035a455c8e053c96fdf568dcd6b8c52c41a6" Oct 29 22:00:06.008854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.008837 4952 scope.go:122] "RemoveContainer" containerID="b6057a55ff29f412fd5ee5c41de5ef3ae38d5732b2824cfdda10fcdfd6aeaa8b" Oct 29 22:00:06.009675 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:06.009648 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4_odh-ai-gateway-infra(e09dc521-2f96-4a1b-aa00-2a28419fdfd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" Oct 29 22:00:06.171927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.171012 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.171927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.171776 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:00:06.209927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.209031 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.263739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.263638 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.264848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.264824 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 22:00:06.308929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.301019 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:00:06.308929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.301074 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:00:06.308929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.303884 4952 generic.go:356] "Generic (PLEG): container finished" podID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerID="97378b9fb52c3d27ecd1d4351fbd1aa38ba48b3c9fec2b3849311e96612bfa65" exitCode=60 Oct 29 22:00:06.361207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.340560 4952 scope.go:122] "RemoveContainer" containerID="97378b9fb52c3d27ecd1d4351fbd1aa38ba48b3c9fec2b3849311e96612bfa65" Oct 29 22:00:06.380937 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:06.367523 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv_odh-ai-gateway-infra(697fcd6a-93d5-40d5-8b49-bbd715b7f7ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" Oct 29 22:00:06.448974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.443989 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" event={"ID":"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca","Type":"ContainerDied","Data":"97378b9fb52c3d27ecd1d4351fbd1aa38ba48b3c9fec2b3849311e96612bfa65"} Oct 29 22:00:06.488795 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.488740 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 29 22:00:06.503126 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.503013 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 22:00:06.534388 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.504860 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:06.534699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.534449 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 22:00:06.623245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.608208 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:00:06.623245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.608284 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:00:06.623245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.622536 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.623245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.623066 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.623884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.623664 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:00:06.623884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.623721 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 22:00:06.691372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.690989 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.703327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.703293 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:06.736248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.736208 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 29 22:00:06.905289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.905257 4952 scope.go:122] "RemoveContainer" containerID="7d4cfd8878a10d5a7ae62bcc999571c96d164930b06c585a28be4d1a8cf62bda" Oct 29 22:00:06.985054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:06.975003 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:00:07.073585 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.065209 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 22:00:07.166987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.154361 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:00:07.194277 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.183429 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.194277 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.183527 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:00:07.224929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.221057 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.224929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.222279 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:00:07.283349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.270624 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:00:07.283349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.270693 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:00:07.500350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.451754 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:07.500350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.456325 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.540106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.517431 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.540106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.528988 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:00:07.540106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.529656 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 22:00:07.540106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.533245 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.756990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.681869 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.756990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.707297 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.756990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.712056 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 22:00:07.756990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.723516 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:00:07.756990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.737078 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:00:07.756990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.737154 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.756990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.737941 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:07.756990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.738224 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 22:00:07.946038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:07.945991 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 22:00:08.057453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.057356 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:00:08.058786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.058762 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 22:00:08.116162 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.116137 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 22:00:08.125632 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.125603 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" event={"ID":"8925c000-d6eb-4bee-beb6-526447d50471","Type":"ContainerStarted","Data":"9d9a2870ae8067d3c9a41c9f036d61652e27bb7aff74caee15ee2a31ebcb312f"} Oct 29 22:00:08.126344 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.126322 4952 scope.go:122] "RemoveContainer" containerID="9d9a2870ae8067d3c9a41c9f036d61652e27bb7aff74caee15ee2a31ebcb312f" Oct 29 22:00:08.126888 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:08.126862 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp_odh-ai-gateway-infra(8925c000-d6eb-4bee-beb6-526447d50471)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" podUID="8925c000-d6eb-4bee-beb6-526447d50471" Oct 29 22:00:08.140811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.140694 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:00:08.140811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.140747 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:00:08.174373 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.174337 4952 generic.go:356] "Generic (PLEG): container finished" podID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerID="cb498fff677404c3ed1c1035e0567a225ab6801d77dcb46ae6746a6415312b13" exitCode=60 Oct 29 22:00:08.174722 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.174695 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" event={"ID":"a552076f-7163-4206-a7b6-ae05b998d9b6","Type":"ContainerDied","Data":"cb498fff677404c3ed1c1035e0567a225ab6801d77dcb46ae6746a6415312b13"} Oct 29 22:00:08.174917 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.174881 4952 scope.go:122] "RemoveContainer" containerID="61d0d838c2a486eca7ac9aa749e5f00a0bca9daa656a4acf6bdaf867ecb7eb3a" Oct 29 22:00:08.206674 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.192049 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:08.241444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.241404 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:00:08.258339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.258315 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 22:00:08.270578 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.258497 4952 scope.go:122] "RemoveContainer" containerID="cb498fff677404c3ed1c1035e0567a225ab6801d77dcb46ae6746a6415312b13" Oct 29 22:00:08.284940 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:08.284764 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc_odh-ai-gateway-infra(a552076f-7163-4206-a7b6-ae05b998d9b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" Oct 29 22:00:08.410937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.407743 4952 scope.go:122] "RemoveContainer" containerID="b6057a55ff29f412fd5ee5c41de5ef3ae38d5732b2824cfdda10fcdfd6aeaa8b" Oct 29 22:00:08.410937 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:08.408968 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4_odh-ai-gateway-infra(e09dc521-2f96-4a1b-aa00-2a28419fdfd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" Oct 29 22:00:08.425926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.421380 4952 scope.go:122] "RemoveContainer" containerID="97378b9fb52c3d27ecd1d4351fbd1aa38ba48b3c9fec2b3849311e96612bfa65" Oct 29 22:00:08.425926 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:08.421636 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv_odh-ai-gateway-infra(697fcd6a-93d5-40d5-8b49-bbd715b7f7ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" Oct 29 22:00:08.425926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.422287 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 22:00:08.468939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.464289 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:08.468939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.464753 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:00:08.501430 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.501401 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 22:00:08.501740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.501723 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:00:08.503561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.503410 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 22:00:08.539260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.522654 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:00:08.570936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.554605 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:08.762974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.762710 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:08.770542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.767052 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:08.930937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.930597 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 22:00:08.955553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.955522 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 22:00:08.957656 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:08.957630 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 22:00:09.181950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:09.181917 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:00:09.182334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:09.182196 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:00:09.184483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:09.184456 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:00:09.214260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:09.214230 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 22:00:09.492992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:09.489402 4952 generic.go:356] "Generic (PLEG): container finished" podID="8925c000-d6eb-4bee-beb6-526447d50471" containerID="9d9a2870ae8067d3c9a41c9f036d61652e27bb7aff74caee15ee2a31ebcb312f" exitCode=60 Oct 29 22:00:09.492992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:09.489491 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" event={"ID":"8925c000-d6eb-4bee-beb6-526447d50471","Type":"ContainerDied","Data":"9d9a2870ae8067d3c9a41c9f036d61652e27bb7aff74caee15ee2a31ebcb312f"} Oct 29 22:00:09.492992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:09.489542 4952 scope.go:122] "RemoveContainer" containerID="646fad0cc6a7b33fe428291835e83c8ef8cf8dce027033eaf06a1bc043bff6a7" Oct 29 22:00:09.492992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:09.489994 4952 scope.go:122] "RemoveContainer" containerID="9d9a2870ae8067d3c9a41c9f036d61652e27bb7aff74caee15ee2a31ebcb312f" Oct 29 22:00:09.492992 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:09.490305 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp_odh-ai-gateway-infra(8925c000-d6eb-4bee-beb6-526447d50471)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" podUID="8925c000-d6eb-4bee-beb6-526447d50471" Oct 29 22:00:09.495417 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:09.494717 4952 scope.go:122] "RemoveContainer" containerID="cb498fff677404c3ed1c1035e0567a225ab6801d77dcb46ae6746a6415312b13" Oct 29 22:00:09.495417 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:09.495011 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc_odh-ai-gateway-infra(a552076f-7163-4206-a7b6-ae05b998d9b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" Oct 29 22:00:10.164338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:10.145273 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:00:10.164338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:10.145336 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:00:10.507533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:10.507457 4952 scope.go:122] "RemoveContainer" containerID="9d9a2870ae8067d3c9a41c9f036d61652e27bb7aff74caee15ee2a31ebcb312f" Oct 29 22:00:10.508019 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:10.507748 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp_odh-ai-gateway-infra(8925c000-d6eb-4bee-beb6-526447d50471)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" podUID="8925c000-d6eb-4bee-beb6-526447d50471" Oct 29 22:00:10.739777 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:10.739726 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:10.937941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:10.937506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:11.154939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:11.153981 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:00:11.154939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:11.154102 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:00:11.154939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:11.154849 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:12.139024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:12.138958 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:00:12.139024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:12.139018 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:00:12.889017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:12.888986 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:13.757170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:13.739156 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:13.757170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:13.740307 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:13.757170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:13.749123 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:00:14.431712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:14.430878 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:14.752309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:14.752204 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:14.847470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:14.838735 4952 scope.go:122] "RemoveContainer" containerID="815de20d306c2030d94da07c355b9854d9508c18c8978e6b22bd6e5c73ec0dd5" Oct 29 22:00:14.871252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:14.871216 4952 scope.go:122] "RemoveContainer" containerID="62451bae29d351d0689fd4f7e61bf4e63e31ef59171a35069d88111084bc0e40" Oct 29 22:00:15.529792 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:15.529757 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:00:15.637006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:15.636966 4952 scope.go:122] "RemoveContainer" containerID="97378b9fb52c3d27ecd1d4351fbd1aa38ba48b3c9fec2b3849311e96612bfa65" Oct 29 22:00:16.433457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.433418 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l"] Oct 29 22:00:16.438770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.438744 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:16.456642 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.456611 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l"] Oct 29 22:00:16.468220 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.463060 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:00:16.513713 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.513677 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d522daa-ctt2l\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:16.513885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.513742 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b99k8\" (UniqueName: \"kubernetes.io/projected/da8a39fb-478f-4a05-b187-a60e82c6d2f2-kube-api-access-b99k8\") pod \"maas-api-revoke-sub-4d522daa-ctt2l\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:16.513885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.513801 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d522daa-ctt2l\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:16.571837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.571349 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:16.615608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.615562 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d522daa-ctt2l\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:16.615806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.615638 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99k8\" (UniqueName: \"kubernetes.io/projected/da8a39fb-478f-4a05-b187-a60e82c6d2f2-kube-api-access-b99k8\") pod \"maas-api-revoke-sub-4d522daa-ctt2l\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:16.615806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.615682 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d522daa-ctt2l\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:16.616370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.616348 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d522daa-ctt2l\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:16.618545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.618518 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d522daa-ctt2l\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:16.639986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.639880 4952 scope.go:122] "RemoveContainer" containerID="b6057a55ff29f412fd5ee5c41de5ef3ae38d5732b2824cfdda10fcdfd6aeaa8b" Oct 29 22:00:16.642367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.641598 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99k8\" (UniqueName: \"kubernetes.io/projected/da8a39fb-478f-4a05-b187-a60e82c6d2f2-kube-api-access-b99k8\") pod \"maas-api-revoke-sub-4d522daa-ctt2l\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:16.670691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.670651 4952 generic.go:356] "Generic (PLEG): container finished" podID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerID="b5f5c774c39ef08d2383a3ea4da2e7eee6e3296607a17bc8cf17b8c0c0bb0840" exitCode=60 Oct 29 22:00:16.673361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.673326 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" event={"ID":"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca","Type":"ContainerDied","Data":"b5f5c774c39ef08d2383a3ea4da2e7eee6e3296607a17bc8cf17b8c0c0bb0840"} Oct 29 22:00:16.673487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.673412 4952 scope.go:122] "RemoveContainer" containerID="97378b9fb52c3d27ecd1d4351fbd1aa38ba48b3c9fec2b3849311e96612bfa65" Oct 29 22:00:16.676551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.676526 4952 scope.go:122] "RemoveContainer" containerID="b5f5c774c39ef08d2383a3ea4da2e7eee6e3296607a17bc8cf17b8c0c0bb0840" Oct 29 22:00:16.677190 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:16.677114 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv_odh-ai-gateway-infra(697fcd6a-93d5-40d5-8b49-bbd715b7f7ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" Oct 29 22:00:16.754427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:16.754390 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:17.073318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:17.073291 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l"] Oct 29 22:00:17.073853 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:00:17.073827 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8a39fb_478f_4a05_b187_a60e82c6d2f2.slice/crio-360ac59459a9881437b3a26c752c7b20ad12f2971f6187d33fe2fc4d2b7e7807 WatchSource:0}: Error finding container 360ac59459a9881437b3a26c752c7b20ad12f2971f6187d33fe2fc4d2b7e7807: Status 404 returned error can't find the container with id 360ac59459a9881437b3a26c752c7b20ad12f2971f6187d33fe2fc4d2b7e7807 Oct 29 22:00:17.636726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:17.636642 4952 scope.go:122] "RemoveContainer" containerID="9d9a2870ae8067d3c9a41c9f036d61652e27bb7aff74caee15ee2a31ebcb312f" Oct 29 22:00:17.636726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:17.636714 4952 scope.go:122] "RemoveContainer" containerID="cb498fff677404c3ed1c1035e0567a225ab6801d77dcb46ae6746a6415312b13" Oct 29 22:00:17.684516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:17.684492 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" event={"ID":"da8a39fb-478f-4a05-b187-a60e82c6d2f2","Type":"ContainerStarted","Data":"f90f564e9ab4db4580abae25e693ce3979957ea0397c1023ad3fa5b74ddbf973"} Oct 29 22:00:17.684632 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:17.684540 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" event={"ID":"da8a39fb-478f-4a05-b187-a60e82c6d2f2","Type":"ContainerStarted","Data":"360ac59459a9881437b3a26c752c7b20ad12f2971f6187d33fe2fc4d2b7e7807"} Oct 29 22:00:17.686894 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:17.686872 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" event={"ID":"e09dc521-2f96-4a1b-aa00-2a28419fdfd2","Type":"ContainerStarted","Data":"b531e27c582084142a3c94360eeefa681fb119edf450ac67e99758c55eb0616e"} Oct 29 22:00:17.729416 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:17.729386 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv"] Oct 29 22:00:17.739655 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:17.739574 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" podStartSLOduration=1.739555426 podStartE2EDuration="1.739555426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:17.726589412 +0000 UTC m=+1083.201177831" watchObservedRunningTime="2026-10-29 22:00:17.739555426 +0000 UTC m=+1083.214143843" Oct 29 22:00:18.139771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.139747 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" Oct 29 22:00:18.239361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.239326 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-maas-api-service-ca\") pod \"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca\" (UID: \"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca\") " Oct 29 22:00:18.239457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.239398 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-kube-api-access-xsclg\" (UniqueName: \"kubernetes.io/projected/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-kube-api-access-xsclg\") pod \"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca\" (UID: \"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca\") " Oct 29 22:00:18.242194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.242153 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-maas-api-service-ca" pod "697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" (UID: "697fcd6a-93d5-40d5-8b49-bbd715b7f7ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:18.244758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.244736 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-kube-api-access-xsclg" pod "697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" (UID: "697fcd6a-93d5-40d5-8b49-bbd715b7f7ca"). InnerVolumeSpecName "kube-api-access-xsclg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:18.347970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.347501 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:18.348932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.348866 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:18.349133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.349101 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsclg\" (UniqueName: \"kubernetes.io/projected/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca-kube-api-access-xsclg\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:18.698522 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.698496 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" Oct 29 22:00:18.698944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.698528 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv" event={"ID":"697fcd6a-93d5-40d5-8b49-bbd715b7f7ca","Type":"ContainerDied","Data":"e746e2f3ed08077258ba8338c85e4be8fdeb9a2ff34b750325e10c18cf00ca8d"} Oct 29 22:00:18.698944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.698595 4952 scope.go:122] "RemoveContainer" containerID="b5f5c774c39ef08d2383a3ea4da2e7eee6e3296607a17bc8cf17b8c0c0bb0840" Oct 29 22:00:18.701149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.701122 4952 generic.go:356] "Generic (PLEG): container finished" podID="da8a39fb-478f-4a05-b187-a60e82c6d2f2" containerID="f90f564e9ab4db4580abae25e693ce3979957ea0397c1023ad3fa5b74ddbf973" exitCode=0 Oct 29 22:00:18.701265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.701224 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" event={"ID":"da8a39fb-478f-4a05-b187-a60e82c6d2f2","Type":"ContainerDied","Data":"f90f564e9ab4db4580abae25e693ce3979957ea0397c1023ad3fa5b74ddbf973"} Oct 29 22:00:18.703813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.703789 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" event={"ID":"8925c000-d6eb-4bee-beb6-526447d50471","Type":"ContainerStarted","Data":"3c0c924dc0b23b424c70f99958d1049c0cf999617af4b5b614b203b05a4efe91"} Oct 29 22:00:18.706247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.706224 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" event={"ID":"a552076f-7163-4206-a7b6-ae05b998d9b6","Type":"ContainerStarted","Data":"67ffdb018d5aae759b498886c517e8b732b31a90a7746c0a09e087d9741e9034"} Oct 29 22:00:18.708526 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.708505 4952 generic.go:356] "Generic (PLEG): container finished" podID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerID="b531e27c582084142a3c94360eeefa681fb119edf450ac67e99758c55eb0616e" exitCode=60 Oct 29 22:00:18.708644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.708566 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" event={"ID":"e09dc521-2f96-4a1b-aa00-2a28419fdfd2","Type":"ContainerDied","Data":"b531e27c582084142a3c94360eeefa681fb119edf450ac67e99758c55eb0616e"} Oct 29 22:00:18.708944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.708895 4952 scope.go:122] "RemoveContainer" containerID="b531e27c582084142a3c94360eeefa681fb119edf450ac67e99758c55eb0616e" Oct 29 22:00:18.709205 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:18.709182 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4_odh-ai-gateway-infra(e09dc521-2f96-4a1b-aa00-2a28419fdfd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" Oct 29 22:00:18.712029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.712006 4952 scope.go:122] "RemoveContainer" containerID="b6057a55ff29f412fd5ee5c41de5ef3ae38d5732b2824cfdda10fcdfd6aeaa8b" Oct 29 22:00:18.780388 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.780358 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4"] Oct 29 22:00:18.804780 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.804743 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv"] Oct 29 22:00:18.816219 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:18.816185 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f8867e7d-29888520-4ltnv"] Oct 29 22:00:19.719400 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.719368 4952 generic.go:356] "Generic (PLEG): container finished" podID="8925c000-d6eb-4bee-beb6-526447d50471" containerID="3c0c924dc0b23b424c70f99958d1049c0cf999617af4b5b614b203b05a4efe91" exitCode=60 Oct 29 22:00:19.719893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.719425 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" event={"ID":"8925c000-d6eb-4bee-beb6-526447d50471","Type":"ContainerDied","Data":"3c0c924dc0b23b424c70f99958d1049c0cf999617af4b5b614b203b05a4efe91"} Oct 29 22:00:19.719893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.719508 4952 scope.go:122] "RemoveContainer" containerID="9d9a2870ae8067d3c9a41c9f036d61652e27bb7aff74caee15ee2a31ebcb312f" Oct 29 22:00:19.720090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.720068 4952 scope.go:122] "RemoveContainer" containerID="3c0c924dc0b23b424c70f99958d1049c0cf999617af4b5b614b203b05a4efe91" Oct 29 22:00:19.720484 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:19.720459 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp_odh-ai-gateway-infra(8925c000-d6eb-4bee-beb6-526447d50471)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" podUID="8925c000-d6eb-4bee-beb6-526447d50471" Oct 29 22:00:19.722137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.722113 4952 generic.go:356] "Generic (PLEG): container finished" podID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerID="67ffdb018d5aae759b498886c517e8b732b31a90a7746c0a09e087d9741e9034" exitCode=60 Oct 29 22:00:19.722232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.722177 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" event={"ID":"a552076f-7163-4206-a7b6-ae05b998d9b6","Type":"ContainerDied","Data":"67ffdb018d5aae759b498886c517e8b732b31a90a7746c0a09e087d9741e9034"} Oct 29 22:00:19.722606 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.722586 4952 scope.go:122] "RemoveContainer" containerID="67ffdb018d5aae759b498886c517e8b732b31a90a7746c0a09e087d9741e9034" Oct 29 22:00:19.723005 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:19.722986 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc_odh-ai-gateway-infra(a552076f-7163-4206-a7b6-ae05b998d9b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" Oct 29 22:00:19.735875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.735855 4952 scope.go:122] "RemoveContainer" containerID="cb498fff677404c3ed1c1035e0567a225ab6801d77dcb46ae6746a6415312b13" Oct 29 22:00:19.867541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.867511 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc"] Oct 29 22:00:19.911852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.911720 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp"] Oct 29 22:00:19.972960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.968307 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" Oct 29 22:00:19.975969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:19.973591 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:20.072916 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.072841 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-openshift-ca\") pod \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " Oct 29 22:00:20.073078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.072977 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-xks-ca\") pod \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " Oct 29 22:00:20.073078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.073036 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8a39fb-478f-4a05-b187-a60e82c6d2f2-kube-api-access-b99k8\" (UniqueName: \"kubernetes.io/projected/da8a39fb-478f-4a05-b187-a60e82c6d2f2-kube-api-access-b99k8\") pod \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\" (UID: \"da8a39fb-478f-4a05-b187-a60e82c6d2f2\") " Oct 29 22:00:20.073078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.073068 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-kube-api-access-k2tsx\" (UniqueName: \"kubernetes.io/projected/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-kube-api-access-k2tsx\") pod \"e09dc521-2f96-4a1b-aa00-2a28419fdfd2\" (UID: \"e09dc521-2f96-4a1b-aa00-2a28419fdfd2\") " Oct 29 22:00:20.073305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.073277 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-openshift-ca" pod "da8a39fb-478f-4a05-b187-a60e82c6d2f2" (UID: "da8a39fb-478f-4a05-b187-a60e82c6d2f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:20.073369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.073326 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-maas-api-service-ca\") pod \"e09dc521-2f96-4a1b-aa00-2a28419fdfd2\" (UID: \"e09dc521-2f96-4a1b-aa00-2a28419fdfd2\") " Oct 29 22:00:20.074047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.074023 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-openshift-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:20.074742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.074486 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-maas-api-service-ca" pod "e09dc521-2f96-4a1b-aa00-2a28419fdfd2" (UID: "e09dc521-2f96-4a1b-aa00-2a28419fdfd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:20.083349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.083320 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-xks-ca" pod "da8a39fb-478f-4a05-b187-a60e82c6d2f2" (UID: "da8a39fb-478f-4a05-b187-a60e82c6d2f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:00:20.083853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.083828 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8a39fb-478f-4a05-b187-a60e82c6d2f2-kube-api-access-b99k8" pod "da8a39fb-478f-4a05-b187-a60e82c6d2f2" (UID: "da8a39fb-478f-4a05-b187-a60e82c6d2f2"). InnerVolumeSpecName "kube-api-access-b99k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:20.091751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.091719 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-kube-api-access-k2tsx" pod "e09dc521-2f96-4a1b-aa00-2a28419fdfd2" (UID: "e09dc521-2f96-4a1b-aa00-2a28419fdfd2"). InnerVolumeSpecName "kube-api-access-k2tsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:20.137664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.137610 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:20.176288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.176258 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:20.176288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.176290 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da8a39fb-478f-4a05-b187-a60e82c6d2f2-maas-api-xks-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:20.176458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.176301 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b99k8\" (UniqueName: \"kubernetes.io/projected/da8a39fb-478f-4a05-b187-a60e82c6d2f2-kube-api-access-b99k8\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:20.176458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.176311 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2tsx\" (UniqueName: \"kubernetes.io/projected/e09dc521-2f96-4a1b-aa00-2a28419fdfd2-kube-api-access-k2tsx\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:20.646580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.646548 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" path="/var/lib/kubelet/pods/697fcd6a-93d5-40d5-8b49-bbd715b7f7ca/volumes" Oct 29 22:00:20.732500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.732469 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" Oct 29 22:00:20.732500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.732497 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l" event={"ID":"da8a39fb-478f-4a05-b187-a60e82c6d2f2","Type":"ContainerDied","Data":"360ac59459a9881437b3a26c752c7b20ad12f2971f6187d33fe2fc4d2b7e7807"} Oct 29 22:00:20.733067 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.732546 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="360ac59459a9881437b3a26c752c7b20ad12f2971f6187d33fe2fc4d2b7e7807" Oct 29 22:00:20.738261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.738237 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" Oct 29 22:00:20.738364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.738271 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4" event={"ID":"e09dc521-2f96-4a1b-aa00-2a28419fdfd2","Type":"ContainerDied","Data":"6eddcc25d151029a3f3fff64c6f3762ec316f3d3436cd9923a814edfd23828f6"} Oct 29 22:00:20.738364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.738336 4952 scope.go:122] "RemoveContainer" containerID="b531e27c582084142a3c94360eeefa681fb119edf450ac67e99758c55eb0616e" Oct 29 22:00:20.829959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.829756 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4"] Oct 29 22:00:20.836166 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.836131 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888520-qmtw4"] Oct 29 22:00:20.972094 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.972069 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" Oct 29 22:00:20.974321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:20.974300 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" Oct 29 22:00:21.087861 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.087835 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a552076f-7163-4206-a7b6-ae05b998d9b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a552076f-7163-4206-a7b6-ae05b998d9b6-maas-api-service-ca\") pod \"a552076f-7163-4206-a7b6-ae05b998d9b6\" (UID: \"a552076f-7163-4206-a7b6-ae05b998d9b6\") " Oct 29 22:00:21.088092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.087914 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8925c000-d6eb-4bee-beb6-526447d50471-kube-api-access-kwkzw\" (UniqueName: \"kubernetes.io/projected/8925c000-d6eb-4bee-beb6-526447d50471-kube-api-access-kwkzw\") pod \"8925c000-d6eb-4bee-beb6-526447d50471\" (UID: \"8925c000-d6eb-4bee-beb6-526447d50471\") " Oct 29 22:00:21.088172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.088111 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8925c000-d6eb-4bee-beb6-526447d50471-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8925c000-d6eb-4bee-beb6-526447d50471-maas-api-service-ca\") pod \"8925c000-d6eb-4bee-beb6-526447d50471\" (UID: \"8925c000-d6eb-4bee-beb6-526447d50471\") " Oct 29 22:00:21.088172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.088162 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a552076f-7163-4206-a7b6-ae05b998d9b6-kube-api-access-8vs8m\" (UniqueName: \"kubernetes.io/projected/a552076f-7163-4206-a7b6-ae05b998d9b6-kube-api-access-8vs8m\") pod \"a552076f-7163-4206-a7b6-ae05b998d9b6\" (UID: \"a552076f-7163-4206-a7b6-ae05b998d9b6\") " Oct 29 22:00:21.088694 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.088668 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8925c000-d6eb-4bee-beb6-526447d50471-maas-api-service-ca" pod "8925c000-d6eb-4bee-beb6-526447d50471" (UID: "8925c000-d6eb-4bee-beb6-526447d50471"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:21.089023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.088998 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a552076f-7163-4206-a7b6-ae05b998d9b6-maas-api-service-ca" pod "a552076f-7163-4206-a7b6-ae05b998d9b6" (UID: "a552076f-7163-4206-a7b6-ae05b998d9b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:00:21.091011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.090986 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a552076f-7163-4206-a7b6-ae05b998d9b6-kube-api-access-8vs8m" pod "a552076f-7163-4206-a7b6-ae05b998d9b6" (UID: "a552076f-7163-4206-a7b6-ae05b998d9b6"). InnerVolumeSpecName "kube-api-access-8vs8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:21.091181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.091156 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8925c000-d6eb-4bee-beb6-526447d50471-kube-api-access-kwkzw" pod "8925c000-d6eb-4bee-beb6-526447d50471" (UID: "8925c000-d6eb-4bee-beb6-526447d50471"). InnerVolumeSpecName "kube-api-access-kwkzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:00:21.190260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.190180 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8925c000-d6eb-4bee-beb6-526447d50471-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:21.190260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.190209 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vs8m\" (UniqueName: \"kubernetes.io/projected/a552076f-7163-4206-a7b6-ae05b998d9b6-kube-api-access-8vs8m\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:21.190260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.190219 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a552076f-7163-4206-a7b6-ae05b998d9b6-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:21.190260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.190228 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwkzw\" (UniqueName: \"kubernetes.io/projected/8925c000-d6eb-4bee-beb6-526447d50471-kube-api-access-kwkzw\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:00:21.747848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.747821 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" Oct 29 22:00:21.748414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.747820 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp" event={"ID":"8925c000-d6eb-4bee-beb6-526447d50471","Type":"ContainerDied","Data":"f98e65580c1bf8405a14c6553492225d4c9cb927ce7e40eb81486d18321a30a6"} Oct 29 22:00:21.748414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.747963 4952 scope.go:122] "RemoveContainer" containerID="3c0c924dc0b23b424c70f99958d1049c0cf999617af4b5b614b203b05a4efe91" Oct 29 22:00:21.749395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.749376 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" Oct 29 22:00:21.749507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.749408 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc" event={"ID":"a552076f-7163-4206-a7b6-ae05b998d9b6","Type":"ContainerDied","Data":"c3e92bba66ada2352281f82d12edfb2ed3e39726deb20cf801cd11daee9ed3dd"} Oct 29 22:00:21.761428 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.761405 4952 scope.go:122] "RemoveContainer" containerID="67ffdb018d5aae759b498886c517e8b732b31a90a7746c0a09e087d9741e9034" Oct 29 22:00:21.802554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.802529 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc"] Oct 29 22:00:21.813701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.813677 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888520-jf5cc"] Oct 29 22:00:21.852436 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.852397 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp"] Oct 29 22:00:21.858648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.858609 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888520-jmmnp"] Oct 29 22:00:21.951462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:21.951422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:22.645749 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:22.645715 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8925c000-d6eb-4bee-beb6-526447d50471" path="/var/lib/kubelet/pods/8925c000-d6eb-4bee-beb6-526447d50471/volumes" Oct 29 22:00:22.646323 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:22.646305 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" path="/var/lib/kubelet/pods/a552076f-7163-4206-a7b6-ae05b998d9b6/volumes" Oct 29 22:00:22.646805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:22.646788 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" path="/var/lib/kubelet/pods/e09dc521-2f96-4a1b-aa00-2a28419fdfd2/volumes" Oct 29 22:00:23.745449 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:23.745408 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:25.624407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:25.624369 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:27.425875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:27.425842 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:28.151536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:28.151496 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:28.830216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:28.830182 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:29.218439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:29.218343 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:29.414932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:29.413210 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:29.903457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:29.903374 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:30.943696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:30.943609 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:32.684132 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:32.684097 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:33.001791 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:33.001695 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:33.185602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:33.185564 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:34.568203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:34.568155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:34.932880 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:34.932846 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:35.175861 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:35.175823 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:36.520808 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:36.520770 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:36.970390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:36.970355 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:38.199677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:38.199616 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:39.958974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:39.952196 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:41.835330 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:41.835287 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:43.931056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:43.931008 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:43.998099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:43.998043 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.187085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:44.186986 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:00:44.429176 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:44.429116 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.488167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:44.488055 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.523040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:44.522991 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.546437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:44.546388 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 22:00:44.690760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:44.690335 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 22:00:44.726964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:44.726919 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 29 22:00:44.784094 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:44.783988 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:44.826211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:44.826148 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 29 22:00:45.007363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.007306 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.008952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.008894 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:00:45.074071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.050191 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.083934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.079283 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.343249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.278898 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:00:45.347032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.347006 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:00:45.493346 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.493303 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.539758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.537994 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.595092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.595006 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 22:00:45.620077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.620049 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:00:45.745945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.741077 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:45.745945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.741645 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 22:00:45.797291 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.797250 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:00:45.805962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:45.805934 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 22:00:46.025462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.024716 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 29 22:00:46.225929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.200132 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 22:00:46.284942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.284808 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.321926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.316086 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.321926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.317535 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 22:00:46.325674 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.323119 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:00:46.380225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.362068 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.403100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.403062 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.528985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.528950 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 22:00:46.671979 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:00:46.664418 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:00:46.757192 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.757148 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.758058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.758030 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:00:46.860112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.856644 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:00:46.950300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.950215 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:00:46.950614 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.950594 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 22:00:46.950843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.950820 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:00:46.951025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:46.951010 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 22:00:47.011815 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:47.011783 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 22:00:47.416097 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:47.416061 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:47.628037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:47.628003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:49.150202 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:49.150162 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:49.590866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:49.587739 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:49.762507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:49.762470 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:51.039819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:51.039780 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:52.784389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:52.784347 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:54.477844 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:54.477791 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:56.092822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:56.092733 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:56.273841 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:56.273782 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:57.853272 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:57.853235 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:00:59.725332 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:00:59.725286 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:01.444072 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:01.444031 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:03.145143 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:03.145105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:04.773260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:04.773224 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:06.456189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:06.456154 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:07.441296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:07.441262 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:08.226284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:08.226248 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:09.949516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:09.949477 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:11.635197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:11.635160 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:13.340808 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:13.340770 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:15.082263 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:15.081418 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:15.528982 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:01:15.528949 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:01:16.826285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:16.826246 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:17.337253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.337218 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh"] Oct 29 22:01:17.338146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338123 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerName="cleanup" Oct 29 22:01:17.338146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338147 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerName="cleanup" Oct 29 22:01:17.338273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338160 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerName="cleanup" Oct 29 22:01:17.338273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338167 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerName="cleanup" Oct 29 22:01:17.338273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338183 4952 cpu_manager.go:414] "containerMap: removing container" podUID="da8a39fb-478f-4a05-b187-a60e82c6d2f2" containerName="revoke-keys" Oct 29 22:01:17.338273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338191 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8a39fb-478f-4a05-b187-a60e82c6d2f2" containerName="revoke-keys" Oct 29 22:01:17.338273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338220 4952 cpu_manager.go:414] "containerMap: removing container" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerName="cleanup" Oct 29 22:01:17.338273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338227 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerName="cleanup" Oct 29 22:01:17.338273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338244 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerName="cleanup" Oct 29 22:01:17.338273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338251 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerName="cleanup" Oct 29 22:01:17.338273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338274 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8925c000-d6eb-4bee-beb6-526447d50471" containerName="cleanup" Oct 29 22:01:17.338521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338281 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8925c000-d6eb-4bee-beb6-526447d50471" containerName="cleanup" Oct 29 22:01:17.338521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338293 4952 cpu_manager.go:414] "containerMap: removing container" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerName="cleanup" Oct 29 22:01:17.338521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338299 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerName="cleanup" Oct 29 22:01:17.338521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338333 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8925c000-d6eb-4bee-beb6-526447d50471" containerName="cleanup" Oct 29 22:01:17.338521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338339 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8925c000-d6eb-4bee-beb6-526447d50471" containerName="cleanup" Oct 29 22:01:17.338521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338352 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8925c000-d6eb-4bee-beb6-526447d50471" containerName="cleanup" Oct 29 22:01:17.338521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338358 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8925c000-d6eb-4bee-beb6-526447d50471" containerName="cleanup" Oct 29 22:01:17.338521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338376 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerName="cleanup" Oct 29 22:01:17.338521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338383 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerName="cleanup" Oct 29 22:01:17.338521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338520 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerName="cleanup" Oct 29 22:01:17.338796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338531 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8925c000-d6eb-4bee-beb6-526447d50471" containerName="cleanup" Oct 29 22:01:17.338796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338540 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8925c000-d6eb-4bee-beb6-526447d50471" containerName="cleanup" Oct 29 22:01:17.338796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338552 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerName="cleanup" Oct 29 22:01:17.338796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338560 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerName="cleanup" Oct 29 22:01:17.338796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338568 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerName="cleanup" Oct 29 22:01:17.338796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338576 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerName="cleanup" Oct 29 22:01:17.338796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338584 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8a39fb-478f-4a05-b187-a60e82c6d2f2" containerName="revoke-keys" Oct 29 22:01:17.338796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338592 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerName="cleanup" Oct 29 22:01:17.338796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338603 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerName="cleanup" Oct 29 22:01:17.338796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.338614 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerName="cleanup" Oct 29 22:01:17.345156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.345134 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.359803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.359769 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03ee8fc-3424-4232-85ad-96a512607f60-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.360057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.359925 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.360057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.359987 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.360172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.360064 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcfs\" (UniqueName: \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-kube-api-access-9bcfs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.360172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.360132 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.360261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.360190 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.360261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.360230 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.360427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.360405 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.360512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.360497 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03ee8fc-3424-4232-85ad-96a512607f60-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.366765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.366739 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh"] Oct 29 22:01:17.462567 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.462529 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03ee8fc-3424-4232-85ad-96a512607f60-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.462734 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.462635 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.462734 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.462690 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.462839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.462739 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcfs\" (UniqueName: \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-kube-api-access-9bcfs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.462839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.462766 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.462839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.462801 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.463070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.462850 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.463070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.462959 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.463070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.463017 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03ee8fc-3424-4232-85ad-96a512607f60-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.463231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.463066 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.463289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.463268 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-certs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.463572 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.463533 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-data\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.463685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.463642 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-credential-socket\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.463827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.463805 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03ee8fc-3424-4232-85ad-96a512607f60-istiod-ca-cert\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.465609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.465588 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-envoy\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.465870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.465849 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03ee8fc-3424-4232-85ad-96a512607f60-istio-podinfo\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.487092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.487065 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-istio-token\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.487184 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.487166 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcfs\" (UniqueName: \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-kube-api-access-9bcfs\") pod \"e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.660877 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.660772 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:17.991380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.991349 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh"] Oct 29 22:01:17.992062 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:01:17.992039 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03ee8fc_3424_4232_85ad_96a512607f60.slice/crio-32c901b3fe1f77c67a1f7de0a07e476ac56dac3b0bcc0fdeb11c66d30bd798e0 WatchSource:0}: Error finding container 32c901b3fe1f77c67a1f7de0a07e476ac56dac3b0bcc0fdeb11c66d30bd798e0: Status 404 returned error can't find the container with id 32c901b3fe1f77c67a1f7de0a07e476ac56dac3b0bcc0fdeb11c66d30bd798e0 Oct 29 22:01:17.994885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.994847 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:01:17.994984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.994949 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:01:17.995028 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:17.994992 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:01:18.435928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:18.427617 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:01:18.674771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:18.674722 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:18.756460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:18.753306 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" event={"ID":"b03ee8fc-3424-4232-85ad-96a512607f60","Type":"ContainerStarted","Data":"3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e"} Oct 29 22:01:18.756460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:18.753389 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" event={"ID":"b03ee8fc-3424-4232-85ad-96a512607f60","Type":"ContainerStarted","Data":"32c901b3fe1f77c67a1f7de0a07e476ac56dac3b0bcc0fdeb11c66d30bd798e0"} Oct 29 22:01:18.875046 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:18.873036 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podStartSLOduration=1.873014602 podStartE2EDuration="1.873014602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:18.83253542 +0000 UTC m=+1144.307123841" watchObservedRunningTime="2026-10-29 22:01:18.873014602 +0000 UTC m=+1144.347603017" Oct 29 22:01:19.662939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:19.662537 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:19.662939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:19.662765 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:01:19.662939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:19.662814 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:01:20.668927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:20.664386 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:01:20.668927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:20.664468 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:01:21.245937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:21.245575 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:21.661929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:21.661388 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:01:21.661929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:21.661496 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:01:22.221335 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:22.221297 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:22.679003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:22.678966 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:01:22.679284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:22.679254 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:01:22.936214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:22.936082 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:22.943030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:22.942980 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:22.943285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:22.942996 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:22.995982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:22.995322 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.035952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.030980 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.185824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.185783 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.210065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.209977 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.332074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.332029 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.370040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.369996 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.437106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.437049 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.535977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.535770 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:23.664295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.664260 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:01:23.664590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.664563 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:01:23.869630 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.869591 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:23.950926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.936850 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 22:01:24.011433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:23.957613 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:01:24.048928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.022205 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 22:01:24.048928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.022313 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 22:01:24.052188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.052152 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:24.285137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.285034 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:24.289337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.289301 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:24.311960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.311924 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:24.318594 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.317835 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:01:24.325332 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.325307 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 22:01:24.397889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.397851 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 22:01:24.428621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.428595 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:01:24.665360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.665326 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:01:24.667451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:24.667416 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:01:25.661786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:25.661730 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 29 22:01:25.662226 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:25.661829 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 29 22:01:25.857789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:25.857519 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:27.413075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:27.413037 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:27.661799 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:27.661761 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:01:27.662015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:27.661826 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:01:27.690979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:27.690192 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:27.691476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:27.691451 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:27.692479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:27.692425 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:01:27.753925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:27.752995 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:27.953380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:27.953297 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:28.164942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:28.164881 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:28.352665 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:28.352633 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:29.535679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:29.535615 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:31.384420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:31.381451 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:32.080931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:32.080758 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:32.274395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:32.274365 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:33.163385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:33.163354 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:33.464109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:33.464011 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:33.770957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:33.770853 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:33.959419 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:33.959384 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:34.191888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:34.191851 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:34.391391 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:34.391339 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:35.674648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:35.674586 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:36.067256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:36.067176 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:36.262380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:36.262126 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:37.650806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:37.650770 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:37.826387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:37.826316 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:39.309263 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:39.309228 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v"] Oct 29 22:01:39.309782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:39.309489 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" podUID="47e708f6-c85d-440a-9713-b18de6189513" containerName="maas-api" containerID="cri-o://30eb004e083e03789ec34058605d161ba8ac320e7bc55b8ee0965be4015aa0f4" gracePeriod=30 Oct 29 22:01:39.448349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:39.448314 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:39.500789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:39.500758 4952 generic.go:356] "Generic (PLEG): container finished" podID="47e708f6-c85d-440a-9713-b18de6189513" containerID="30eb004e083e03789ec34058605d161ba8ac320e7bc55b8ee0965be4015aa0f4" exitCode=0 Oct 29 22:01:39.500973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:39.500818 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" event={"ID":"47e708f6-c85d-440a-9713-b18de6189513","Type":"ContainerDied","Data":"30eb004e083e03789ec34058605d161ba8ac320e7bc55b8ee0965be4015aa0f4"} Oct 29 22:01:39.758948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:39.730273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:40.026953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.026138 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 22:01:40.186723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.186693 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e708f6-c85d-440a-9713-b18de6189513-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e708f6-c85d-440a-9713-b18de6189513-openshift-service-ca\") pod \"47e708f6-c85d-440a-9713-b18de6189513\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " Oct 29 22:01:40.187057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.186870 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-metrics-certs\") pod \"47e708f6-c85d-440a-9713-b18de6189513\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " Oct 29 22:01:40.187123 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.187056 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-maas-api-tls\") pod \"47e708f6-c85d-440a-9713-b18de6189513\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " Oct 29 22:01:40.187168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.187131 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e708f6-c85d-440a-9713-b18de6189513-kube-api-access-25pfz\" (UniqueName: \"kubernetes.io/projected/47e708f6-c85d-440a-9713-b18de6189513-kube-api-access-25pfz\") pod \"47e708f6-c85d-440a-9713-b18de6189513\" (UID: \"47e708f6-c85d-440a-9713-b18de6189513\") " Oct 29 22:01:40.190845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.189068 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e708f6-c85d-440a-9713-b18de6189513-openshift-service-ca" pod "47e708f6-c85d-440a-9713-b18de6189513" (UID: "47e708f6-c85d-440a-9713-b18de6189513"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:01:40.190845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.189950 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-metrics-certs" pod "47e708f6-c85d-440a-9713-b18de6189513" (UID: "47e708f6-c85d-440a-9713-b18de6189513"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:01:40.199140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.199111 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e708f6-c85d-440a-9713-b18de6189513-kube-api-access-25pfz" pod "47e708f6-c85d-440a-9713-b18de6189513" (UID: "47e708f6-c85d-440a-9713-b18de6189513"). InnerVolumeSpecName "kube-api-access-25pfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:01:40.200780 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.200750 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-maas-api-tls" pod "47e708f6-c85d-440a-9713-b18de6189513" (UID: "47e708f6-c85d-440a-9713-b18de6189513"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:01:40.289667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.289577 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25pfz\" (UniqueName: \"kubernetes.io/projected/47e708f6-c85d-440a-9713-b18de6189513-kube-api-access-25pfz\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:01:40.289667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.289609 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e708f6-c85d-440a-9713-b18de6189513-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:01:40.289667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.289624 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:01:40.289667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.289635 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47e708f6-c85d-440a-9713-b18de6189513-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:01:40.314818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.314664 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:40.491894 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.491844 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:40.509972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.509942 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" event={"ID":"47e708f6-c85d-440a-9713-b18de6189513","Type":"ContainerDied","Data":"ed6fce51a3971e81cd43fcf72f91de1659cad4d59f1cea34213e8808dd404577"} Oct 29 22:01:40.509972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.509957 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v" Oct 29 22:01:40.510200 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.510024 4952 scope.go:122] "RemoveContainer" containerID="30eb004e083e03789ec34058605d161ba8ac320e7bc55b8ee0965be4015aa0f4" Oct 29 22:01:40.569675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.569593 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v"] Oct 29 22:01:40.575828 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.575800 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a116a2-7b49694f97-9f27v"] Oct 29 22:01:40.651206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.651173 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e708f6-c85d-440a-9713-b18de6189513" path="/var/lib/kubelet/pods/47e708f6-c85d-440a-9713-b18de6189513/volumes" Oct 29 22:01:40.873554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:40.873524 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:41.073678 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.073632 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:41.381233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.381192 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns"] Oct 29 22:01:41.382033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382013 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerName="cleanup" Oct 29 22:01:41.382033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382033 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a552076f-7163-4206-a7b6-ae05b998d9b6" containerName="cleanup" Oct 29 22:01:41.382190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382058 4952 cpu_manager.go:414] "containerMap: removing container" podUID="47e708f6-c85d-440a-9713-b18de6189513" containerName="maas-api" Oct 29 22:01:41.382190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382066 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e708f6-c85d-440a-9713-b18de6189513" containerName="maas-api" Oct 29 22:01:41.382190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382090 4952 cpu_manager.go:414] "containerMap: removing container" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerName="cleanup" Oct 29 22:01:41.382190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382097 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerName="cleanup" Oct 29 22:01:41.382190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382111 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerName="cleanup" Oct 29 22:01:41.382190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382119 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09dc521-2f96-4a1b-aa00-2a28419fdfd2" containerName="cleanup" Oct 29 22:01:41.382476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382312 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="697fcd6a-93d5-40d5-8b49-bbd715b7f7ca" containerName="cleanup" Oct 29 22:01:41.382476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382330 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8925c000-d6eb-4bee-beb6-526447d50471" containerName="cleanup" Oct 29 22:01:41.382476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.382343 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e708f6-c85d-440a-9713-b18de6189513" containerName="maas-api" Oct 29 22:01:41.387614 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.387592 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.430925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.423987 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns"] Oct 29 22:01:41.431117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.431087 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fcce6b-openshift-default-dockercfg-clqkv\"" Oct 29 22:01:41.492917 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.492504 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:41.507373 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.507335 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-credential-socket\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.507529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.507449 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-data\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.507608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.507580 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istiod-ca-cert\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.507666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.507646 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-socket\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.507796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.507779 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-token\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.507879 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.507866 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-envoy\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.508058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.508045 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-podinfo\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.508187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.508166 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6s7\" (UniqueName: \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-kube-api-access-jq6s7\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.508233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.508207 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-certs\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.610224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.610189 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-token\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.610503 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.610485 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-envoy\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.618128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.618098 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-podinfo\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.618351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.618335 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6s7\" (UniqueName: \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-kube-api-access-jq6s7\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.633977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.618451 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-certs\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.634818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.634786 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-credential-socket\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.635155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.635130 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-data\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.635415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.635397 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istiod-ca-cert\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.635544 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.635529 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-socket\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.638818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.636167 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-socket\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.639522 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.639500 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-certs\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.639947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.639928 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-credential-socket\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.640323 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.640304 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-data\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.641061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.641041 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istiod-ca-cert\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.661927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.659864 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-envoy\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.661927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.660759 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-token\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.679994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.669960 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:41.679994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.672483 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-podinfo\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.679994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.673973 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6s7\" (UniqueName: \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-kube-api-access-jq6s7\") pod \"e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:41.711925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:41.706183 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:42.120192 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:01:42.120158 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925f097f_4d45_4843_b5f6_ef6412b3dfa9.slice/crio-1746eb05bbfd51eccd839a3d0a450f069ee1b6d7a6b43a0ddf4e84ccc8f6dabb WatchSource:0}: Error finding container 1746eb05bbfd51eccd839a3d0a450f069ee1b6d7a6b43a0ddf4e84ccc8f6dabb: Status 404 returned error can't find the container with id 1746eb05bbfd51eccd839a3d0a450f069ee1b6d7a6b43a0ddf4e84ccc8f6dabb Oct 29 22:01:42.124159 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:42.124120 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:01:42.124277 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:42.124209 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:01:42.124277 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:42.124257 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:01:42.126240 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:42.125331 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns"] Oct 29 22:01:42.419411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:42.419129 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:42.537076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:42.536538 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" event={"ID":"925f097f-4d45-4843-b5f6-ef6412b3dfa9","Type":"ContainerStarted","Data":"009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5"} Oct 29 22:01:42.537076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:42.536606 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" event={"ID":"925f097f-4d45-4843-b5f6-ef6412b3dfa9","Type":"ContainerStarted","Data":"1746eb05bbfd51eccd839a3d0a450f069ee1b6d7a6b43a0ddf4e84ccc8f6dabb"} Oct 29 22:01:42.600231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:42.600166 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" podStartSLOduration=1.60014678 podStartE2EDuration="1.60014678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:42.598955975 +0000 UTC m=+1168.073544392" watchObservedRunningTime="2026-10-29 22:01:42.60014678 +0000 UTC m=+1168.074735198" Oct 29 22:01:42.715007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:42.710020 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:43.713691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:43.713656 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:44.293176 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:44.293131 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:44.568187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:44.568108 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:44.569058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:44.569036 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:01:44.961300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:44.961263 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv"] Oct 29 22:01:44.966256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:44.966222 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:44.982357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:44.982334 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f8867e7d-openshift-default-dockercfg-s5lcp\"" Oct 29 22:01:44.999641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:44.999594 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv"] Oct 29 22:01:45.115526 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.115491 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6727\" (UniqueName: \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-kube-api-access-h6727\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.115735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.115560 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-envoy\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.115781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.115758 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-istio-token\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.115855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.115832 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-credential-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.115979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.115958 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6128724b-db39-4b47-8976-85a9756882a1-istio-podinfo\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.116058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.116042 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.116141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.116118 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-data\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.116198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.116149 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-certs\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.116257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.116223 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6128724b-db39-4b47-8976-85a9756882a1-istiod-ca-cert\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.184180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.184143 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:45.217599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.217524 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-istio-token\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.217599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.217583 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-credential-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.217806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.217623 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6128724b-db39-4b47-8976-85a9756882a1-istio-podinfo\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.217806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.217673 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.217806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.217730 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-data\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.217806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.217756 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-certs\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.218043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.217854 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6128724b-db39-4b47-8976-85a9756882a1-istiod-ca-cert\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.218043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.218002 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6727\" (UniqueName: \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-kube-api-access-h6727\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.218155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.218073 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-credential-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.218155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.218080 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-envoy\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.218254 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.218160 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-socket\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.218311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.218245 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-data\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.218708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.218684 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-certs\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.218844 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.218705 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6128724b-db39-4b47-8976-85a9756882a1-istiod-ca-cert\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.220678 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.220652 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-envoy\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.220857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.220839 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6128724b-db39-4b47-8976-85a9756882a1-istio-podinfo\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.251493 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.251458 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6727\" (UniqueName: \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-kube-api-access-h6727\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.252579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.252550 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-istio-token\") pod \"e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.283566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.283538 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:45.406539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.406504 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:45.533065 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:01:45.532984 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:01:45.620662 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:01:45.620623 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6128724b_db39_4b47_8976_85a9756882a1.slice/crio-317b07a270e8795acb0af61be462ae40dbaf0b86b73fd298b99a5fc32e8e8029 WatchSource:0}: Error finding container 317b07a270e8795acb0af61be462ae40dbaf0b86b73fd298b99a5fc32e8e8029: Status 404 returned error can't find the container with id 317b07a270e8795acb0af61be462ae40dbaf0b86b73fd298b99a5fc32e8e8029 Oct 29 22:01:45.620790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.620740 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv"] Oct 29 22:01:45.623554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.623519 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:01:45.623682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.623580 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:01:45.623682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:45.623616 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:01:46.243929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:46.238606 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:46.610793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:46.610663 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" event={"ID":"6128724b-db39-4b47-8976-85a9756882a1","Type":"ContainerStarted","Data":"acbfe18d1e9843ce50445dceac720520ddfc1ec356df6e56a9df1f37ce8ce9de"} Oct 29 22:01:46.610793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:46.610757 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" event={"ID":"6128724b-db39-4b47-8976-85a9756882a1","Type":"ContainerStarted","Data":"317b07a270e8795acb0af61be462ae40dbaf0b86b73fd298b99a5fc32e8e8029"} Oct 29 22:01:46.693547 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:46.693483 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" podStartSLOduration=2.693468678 podStartE2EDuration="2.693468678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:46.681146574 +0000 UTC m=+1172.155734992" watchObservedRunningTime="2026-10-29 22:01:46.693468678 +0000 UTC m=+1172.168057093" Oct 29 22:01:46.969978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:46.965929 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:47.163972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:47.163850 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:47.284077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:47.283997 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:47.285554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:47.285522 4952 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:01:47.285681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:47.285571 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" podUID="6128724b-db39-4b47-8976-85a9756882a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:01:47.452142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:47.451514 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:48.284738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:48.284700 4952 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:01:48.285151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:48.284760 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" podUID="6128724b-db39-4b47-8976-85a9756882a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:01:48.666934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:48.662964 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:49.289929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:49.289307 4952 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:01:49.289929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:49.289377 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" podUID="6128724b-db39-4b47-8976-85a9756882a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:01:50.284590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:50.284554 4952 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:01:50.284762 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:50.284612 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" podUID="6128724b-db39-4b47-8976-85a9756882a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:01:50.654707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:50.653942 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:50.841924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:50.841476 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:51.284543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:51.284508 4952 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:01:51.284755 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:51.284556 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" podUID="6128724b-db39-4b47-8976-85a9756882a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:01:52.284648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:52.284615 4952 patch_prober.go:29] interesting pod/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:01:52.285109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:52.284682 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" podUID="6128724b-db39-4b47-8976-85a9756882a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:01:52.723506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:52.723459 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:53.288359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:53.288328 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:53.288840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:53.288807 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:53.289480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:53.289460 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:01:54.439357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:54.439047 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:56.191004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:56.190969 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:56.446145 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:56.446053 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:58.045676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:58.045633 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:01:59.760629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:01:59.760594 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:00.134168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:00.134114 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:00.574803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:00.574769 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:01.359356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.359316 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h"] Oct 29 22:02:01.364820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.364800 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.384148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.384120 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5fd295-serving-cert\"" Oct 29 22:02:01.384580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.384550 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h"] Oct 29 22:02:01.486549 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.486517 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.486735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.486569 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-metrics-certs\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.486783 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.486729 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9xbh\" (UniqueName: \"kubernetes.io/projected/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-kube-api-access-n9xbh\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.486927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.486890 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.502698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.502670 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:01.589702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.589653 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.589900 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.589741 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-metrics-certs\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.589900 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.589880 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9xbh\" (UniqueName: \"kubernetes.io/projected/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-kube-api-access-n9xbh\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.590449 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.590425 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.590545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.590494 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.594266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.594240 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-metrics-certs\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.596033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.595889 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.614746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.614673 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9xbh\" (UniqueName: \"kubernetes.io/projected/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-kube-api-access-n9xbh\") pod \"maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.683546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.683511 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:01.686424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.686401 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:01.858006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:01.857958 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:02.001605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:02.001579 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h"] Oct 29 22:02:02.001955 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:02:02.001932 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594282a4_47b3_425b_8f9e_1ffac4f2cdc6.slice/crio-140ad8505466281d3830916130ae68755de1635e4aabe0169854f9af4230ec8a WatchSource:0}: Error finding container 140ad8505466281d3830916130ae68755de1635e4aabe0169854f9af4230ec8a: Status 404 returned error can't find the container with id 140ad8505466281d3830916130ae68755de1635e4aabe0169854f9af4230ec8a Oct 29 22:02:02.865150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:02.863700 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" event={"ID":"594282a4-47b3-425b-8f9e-1ffac4f2cdc6","Type":"ContainerStarted","Data":"a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b"} Oct 29 22:02:02.865150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:02.863781 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" event={"ID":"594282a4-47b3-425b-8f9e-1ffac4f2cdc6","Type":"ContainerStarted","Data":"140ad8505466281d3830916130ae68755de1635e4aabe0169854f9af4230ec8a"} Oct 29 22:02:02.865150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:02.865086 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:02.916629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:02.916570 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" podStartSLOduration=1.598016892 podStartE2EDuration="1.916555553s" totalImagesPullingTime="318.538661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:02.914779422 +0000 UTC m=+1188.389367840" watchObservedRunningTime="2026-10-29 22:02:02.916555553 +0000 UTC m=+1188.391143963" Oct 29 22:02:03.368476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:03.368446 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:05.262404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:05.262356 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:07.139891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:07.139852 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:07.553684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:07.553600 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:07.720204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:07.720172 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:08.605530 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:08.605494 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:09.052769 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:09.052680 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:09.251298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:09.251262 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:09.889465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:09.889428 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:02:11.176109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:11.176073 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:12.900672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:12.900634 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:14.585852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:14.585806 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:14.998060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:14.998030 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:02:15.001498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:15.001473 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:02:15.420206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:15.420175 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:15.535255 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:02:15.535213 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:02:15.614675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:15.614643 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:15.807488 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:15.807398 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:16.407937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:16.407885 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:18.055741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:18.055703 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:19.180476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:19.180435 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:02:19.856938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:19.856818 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:21.986928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:21.986692 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:23.454394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:23.454356 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:24.204932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:24.203656 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:24.381186 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:24.381147 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:24.561806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:24.561687 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:24.949346 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:24.949213 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:25.391723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:25.391575 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:26.351079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:26.350834 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:27.086930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:27.083755 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:27.290440 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:27.290404 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:27.459057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:27.458955 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:27.851090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:27.851052 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:28.840097 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:28.840059 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:29.021425 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:29.021389 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:29.270788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:29.270698 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:31.269262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:31.263258 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:31.454386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:31.454313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:33.049838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:33.049803 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:33.262729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:33.262696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:34.709529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:34.709492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:35.269412 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:35.269378 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:35.452067 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:35.452034 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:35.662746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:35.662707 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:35.956258 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:35.956149 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:36.141271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:36.141241 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:36.334922 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:36.332193 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:36.646997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:36.646105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:36.821377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:36.821331 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:37.361153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:37.361118 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:37.541982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:37.539885 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:39.486670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:39.486637 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:41.273445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:41.273407 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:41.443222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:41.443190 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:41.652599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:41.652565 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:43.220857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:43.220805 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:43.408962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:43.408874 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:43.663278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:43.663243 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:43.842261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:43.842227 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:45.169745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:45.168374 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:45.528569 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:02:45.528476 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:02:46.421408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:46.421366 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:46.609215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:46.609185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:47.308780 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:47.308734 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:47.849402 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:47.849368 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:48.100579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:48.100481 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:48.278172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:48.278139 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:48.458659 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:48.458577 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:49.997594 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:49.997556 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:51.226567 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.226525 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:51.403550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.403488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:51.581532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.581496 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv"] Oct 29 22:02:51.592545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.592513 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:02:51.616446 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.616414 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5fd295\"" Oct 29 22:02:51.654552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.638274 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5fd295-dockercfg-w5nlp\"" Oct 29 22:02:51.654552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.640667 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv"] Oct 29 22:02:51.719176 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.719137 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ad5968-00d3-4048-a03d-7e5168abd368-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv\" (UID: \"b6ad5968-00d3-4048-a03d-7e5168abd368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:02:51.719383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.719338 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4gjz\" (UniqueName: \"kubernetes.io/projected/b6ad5968-00d3-4048-a03d-7e5168abd368-kube-api-access-q4gjz\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv\" (UID: \"b6ad5968-00d3-4048-a03d-7e5168abd368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:02:51.822086 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.822045 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ad5968-00d3-4048-a03d-7e5168abd368-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv\" (UID: \"b6ad5968-00d3-4048-a03d-7e5168abd368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:02:51.822278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.822187 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4gjz\" (UniqueName: \"kubernetes.io/projected/b6ad5968-00d3-4048-a03d-7e5168abd368-kube-api-access-q4gjz\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv\" (UID: \"b6ad5968-00d3-4048-a03d-7e5168abd368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:02:51.822787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.822761 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ad5968-00d3-4048-a03d-7e5168abd368-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv\" (UID: \"b6ad5968-00d3-4048-a03d-7e5168abd368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:02:51.853456 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.853375 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4gjz\" (UniqueName: \"kubernetes.io/projected/b6ad5968-00d3-4048-a03d-7e5168abd368-kube-api-access-q4gjz\") pod \"payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv\" (UID: \"b6ad5968-00d3-4048-a03d-7e5168abd368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:02:51.960741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:51.960700 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:02:52.311219 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:52.311106 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:52.337381 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:52.337349 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv"] Oct 29 22:02:52.339154 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:02:52.339110 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ad5968_00d3_4048_a03d_7e5168abd368.slice/crio-df5c4f0c9d9c20f8f36c985721565ac48ceee2f2839add29306bdf7d47d448cf WatchSource:0}: Error finding container df5c4f0c9d9c20f8f36c985721565ac48ceee2f2839add29306bdf7d47d448cf: Status 404 returned error can't find the container with id df5c4f0c9d9c20f8f36c985721565ac48ceee2f2839add29306bdf7d47d448cf Oct 29 22:02:52.486287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:52.486236 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:52.753593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:52.753557 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" event={"ID":"b6ad5968-00d3-4048-a03d-7e5168abd368","Type":"ContainerStarted","Data":"df5c4f0c9d9c20f8f36c985721565ac48ceee2f2839add29306bdf7d47d448cf"} Oct 29 22:02:53.756889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:53.756849 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:53.763965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:53.763931 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" event={"ID":"b6ad5968-00d3-4048-a03d-7e5168abd368","Type":"ContainerStarted","Data":"3a74a8fc58196b948e4c8bbdd75ed94ab8faf8f04b86ce6df3d3634d8cdb9588"} Oct 29 22:02:53.764142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:53.764059 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:02:53.810172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:53.810118 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" podStartSLOduration=1.578873614 podStartE2EDuration="2.810103174s" totalImagesPullingTime="1.23122956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:53.805133721 +0000 UTC m=+1239.279722139" watchObservedRunningTime="2026-10-29 22:02:53.810103174 +0000 UTC m=+1239.284691590" Oct 29 22:02:53.937891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:53.935124 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:54.243000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:54.242860 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:55.771404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.771366 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt"] Oct 29 22:02:55.776321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.776296 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.785659 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.785634 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-metrics-certs\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.785770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.785713 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522ed822-e84d-4351-83de-1c123dc4de04-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.785770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.785765 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gscm4\" (UniqueName: \"kubernetes.io/projected/522ed822-e84d-4351-83de-1c123dc4de04-kube-api-access-gscm4\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.786017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.785994 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.796236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.796176 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt"] Oct 29 22:02:55.796386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.796237 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d9f8c0-serving-cert\"" Oct 29 22:02:55.887406 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.887353 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-metrics-certs\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.887545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.887439 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522ed822-e84d-4351-83de-1c123dc4de04-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.887545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.887472 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gscm4\" (UniqueName: \"kubernetes.io/projected/522ed822-e84d-4351-83de-1c123dc4de04-kube-api-access-gscm4\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.887735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.887711 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.887931 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:02:55.887875 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-serving-cert: secret "maas-api-e2e-ipp-b-d9f8c0-serving-cert" not found Oct 29 22:02:55.888023 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:02:55.887991 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-maas-api-tls podName:522ed822-e84d-4351-83de-1c123dc4de04 nodeName:}" failed. No retries permitted until 2026-10-29 22:02:56.387966055 +0000 UTC m=+1241.862554467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-maas-api-tls") pod "maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" (UID: "522ed822-e84d-4351-83de-1c123dc4de04") : secret "maas-api-e2e-ipp-b-d9f8c0-serving-cert" not found Oct 29 22:02:55.888288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.888270 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522ed822-e84d-4351-83de-1c123dc4de04-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.889929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.889895 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-metrics-certs\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.913349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.913324 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gscm4\" (UniqueName: \"kubernetes.io/projected/522ed822-e84d-4351-83de-1c123dc4de04-kube-api-access-gscm4\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:55.919641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:55.919540 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:56.090862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:56.090831 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:56.257537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:56.257495 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:56.404511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:56.404425 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:56.407990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:56.407962 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:56.692227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:56.691855 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:57.165089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:57.164305 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt"] Oct 29 22:02:57.800383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:57.800351 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" event={"ID":"522ed822-e84d-4351-83de-1c123dc4de04","Type":"ContainerStarted","Data":"1026b364921ddb963832f875c41d194e609b0559fc6542f66792368d6b2c118b"} Oct 29 22:02:57.800600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:57.800407 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" event={"ID":"522ed822-e84d-4351-83de-1c123dc4de04","Type":"ContainerStarted","Data":"6dbc70a6b9797731fab2e4e3416579b8f3a1447322a63ebdc58974dfd62de4f6"} Oct 29 22:02:57.800600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:57.800521 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:02:57.851778 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:57.851707 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" podStartSLOduration=2.366097979 podStartE2EDuration="2.851686595s" totalImagesPullingTime="485.588616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:57.848724127 +0000 UTC m=+1243.323312545" watchObservedRunningTime="2026-10-29 22:02:57.851686595 +0000 UTC m=+1243.326275013" Oct 29 22:02:57.944073 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:57.944037 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:59.678402 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:59.678361 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:02:59.851161 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:02:59.851123 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:00.595813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:00.595780 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:00.802580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:00.802537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:01.808179 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:01.808135 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:03.421515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:03.421479 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:03.814276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:03.814201 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:03:04.458488 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:04.458455 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:03:04.782185 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:04.782094 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:03:05.059004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.058895 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5"] Oct 29 22:03:05.064975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.064946 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.081441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.081409 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-296d19-serving-cert\"" Oct 29 22:03:05.087363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.084461 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5"] Oct 29 22:03:05.178932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.178878 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24h6w\" (UniqueName: \"kubernetes.io/projected/9dc98065-570b-41af-be23-5888293bda1f-kube-api-access-24h6w\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.179095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.179035 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc98065-570b-41af-be23-5888293bda1f-openshift-service-ca\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.179136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.179102 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-metrics-certs\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.179250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.179231 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-maas-api-tls\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.209088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.209053 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:05.280881 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.280834 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc98065-570b-41af-be23-5888293bda1f-openshift-service-ca\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.281077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.280979 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-metrics-certs\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.281152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.281087 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-maas-api-tls\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.281279 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.281262 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24h6w\" (UniqueName: \"kubernetes.io/projected/9dc98065-570b-41af-be23-5888293bda1f-kube-api-access-24h6w\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.281649 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.281611 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc98065-570b-41af-be23-5888293bda1f-openshift-service-ca\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.283673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.283652 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-metrics-certs\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.283786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.283683 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-maas-api-tls\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.315100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.310577 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24h6w\" (UniqueName: \"kubernetes.io/projected/9dc98065-570b-41af-be23-5888293bda1f-kube-api-access-24h6w\") pod \"maas-api-e2e-api-b-296d19-856b97786f-n7mz5\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:05.378988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.378953 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:05.386928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:05.385263 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:06.003833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:06.002575 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:06.130816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:06.130732 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5"] Oct 29 22:03:06.164174 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:03:06.164136 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc98065_570b_41af_be23_5888293bda1f.slice/crio-c104d948ba18e3ffb9431b377b97539bff6fc10302323b8278e86da05669a9e6 WatchSource:0}: Error finding container c104d948ba18e3ffb9431b377b97539bff6fc10302323b8278e86da05669a9e6: Status 404 returned error can't find the container with id c104d948ba18e3ffb9431b377b97539bff6fc10302323b8278e86da05669a9e6 Oct 29 22:03:06.731144 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:06.731109 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:06.914785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:06.914719 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:06.954930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:06.953829 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" event={"ID":"9dc98065-570b-41af-be23-5888293bda1f","Type":"ContainerStarted","Data":"c104d948ba18e3ffb9431b377b97539bff6fc10302323b8278e86da05669a9e6"} Oct 29 22:03:07.653742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:07.649119 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:07.832507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:07.832476 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:07.973007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:07.972918 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" event={"ID":"9dc98065-570b-41af-be23-5888293bda1f","Type":"ContainerStarted","Data":"094faf344b24901b23b8acead9cf312d1742691f4d88ec70e27b8f7066d57a26"} Oct 29 22:03:07.974452 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:07.974430 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:08.019469 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:08.019416 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" podStartSLOduration=2.21284699 podStartE2EDuration="3.019402667s" totalImagesPullingTime="806.555677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:08.017681482 +0000 UTC m=+1253.492269917" watchObservedRunningTime="2026-10-29 22:03:08.019402667 +0000 UTC m=+1253.493991120" Oct 29 22:03:09.053773 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.053732 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:09.391098 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.391069 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4"] Oct 29 22:03:09.505987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.504116 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4"] Oct 29 22:03:09.506222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.504563 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.540815 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.540782 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-442dc2-openshift-default-dockercfg-8vjpc\"" Oct 29 22:03:09.608932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.595113 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-envoy\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.608932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.595218 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-data\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.608932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.595310 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/184e1774-f710-4a0c-a816-af20d9032d86-istiod-ca-cert\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.608932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.595401 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-socket\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.608932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.595456 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/184e1774-f710-4a0c-a816-af20d9032d86-istio-podinfo\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.608932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.595563 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-istio-token\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.608932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.595654 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-certs\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.608932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.595775 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqnzl\" (UniqueName: \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-kube-api-access-kqnzl\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.608932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.595813 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-credential-socket\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.608932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.596759 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.705509 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqnzl\" (UniqueName: \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-kube-api-access-kqnzl\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.705568 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-credential-socket\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.705648 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-envoy\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.705703 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-data\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.705771 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/184e1774-f710-4a0c-a816-af20d9032d86-istiod-ca-cert\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.705838 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-socket\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.705883 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/184e1774-f710-4a0c-a816-af20d9032d86-istio-podinfo\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.705976 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-istio-token\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.706029 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-certs\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.706671 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-certs\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.707794 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-data\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.708450 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-credential-socket\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.711325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.709093 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-socket\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.731928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.723495 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/184e1774-f710-4a0c-a816-af20d9032d86-istiod-ca-cert\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.806465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.795696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:09.845633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.816659 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-istio-token\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.854087 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.847244 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-envoy\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.892729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.887617 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/184e1774-f710-4a0c-a816-af20d9032d86-istio-podinfo\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:09.964801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:09.964722 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqnzl\" (UniqueName: \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-kube-api-access-kqnzl\") pod \"e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:10.202393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:10.202357 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:10.574951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:10.574651 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4"] Oct 29 22:03:10.579017 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:03:10.578982 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184e1774_f710_4a0c_a816_af20d9032d86.slice/crio-d48892df537015dff12a40e67537440a61c11c4ba40c5fb6a20df47877f500c4 WatchSource:0}: Error finding container d48892df537015dff12a40e67537440a61c11c4ba40c5fb6a20df47877f500c4: Status 404 returned error can't find the container with id d48892df537015dff12a40e67537440a61c11c4ba40c5fb6a20df47877f500c4 Oct 29 22:03:10.582692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:10.582652 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:03:10.582788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:10.582741 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:03:10.582839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:10.582786 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:03:11.020542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:11.020434 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" event={"ID":"184e1774-f710-4a0c-a816-af20d9032d86","Type":"ContainerStarted","Data":"4d532fb60a6fa0b8433c1b99a39cd7f4efe3de1df385498cd93ec4eff11ff377"} Oct 29 22:03:11.020542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:11.020509 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" event={"ID":"184e1774-f710-4a0c-a816-af20d9032d86","Type":"ContainerStarted","Data":"d48892df537015dff12a40e67537440a61c11c4ba40c5fb6a20df47877f500c4"} Oct 29 22:03:11.111566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:11.111535 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:11.212926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:11.207829 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:11.212926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:11.208020 4952 patch_prober.go:29] interesting pod/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:03:11.212926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:11.208070 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 29 22:03:11.335694 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:11.335651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:12.209897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:12.209868 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:12.269641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:12.269590 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" podStartSLOduration=3.269575281 podStartE2EDuration="3.269575281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:11.078155557 +0000 UTC m=+1256.552743972" watchObservedRunningTime="2026-10-29 22:03:12.269575281 +0000 UTC m=+1257.744163698" Oct 29 22:03:12.415768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:12.415681 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:12.984547 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:12.984511 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:13.073322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:13.073288 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:13.074776 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:13.074623 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:03:13.178923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:13.178762 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:13.786954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:13.784881 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:13.962297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:13.962260 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:14.892835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:14.892790 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:15.016651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:15.016618 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:03:15.075528 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:15.075434 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:15.249794 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:15.249714 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:15.569393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:15.569306 4952 patch_prober.go:29] interesting pod/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:03:15.569579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:15.569382 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:03:15.584920 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:03:15.584868 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:03:16.141455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:16.141424 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:16.360957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:16.338686 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:17.057712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:17.057677 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:18.119352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:18.119307 4952 generic.go:356] "Generic (PLEG): container finished" podID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerID="009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5" exitCode=0 Oct 29 22:03:18.119814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:18.119386 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" event={"ID":"925f097f-4d45-4843-b5f6-ef6412b3dfa9","Type":"ContainerDied","Data":"009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5"} Oct 29 22:03:18.120033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:18.120015 4952 scope.go:122] "RemoveContainer" containerID="009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5" Oct 29 22:03:18.121866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:18.121832 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:03:18.121994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:18.121936 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:03:18.121994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:18.121988 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:03:18.872885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:18.872830 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:19.159468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:19.159373 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" event={"ID":"925f097f-4d45-4843-b5f6-ef6412b3dfa9","Type":"ContainerStarted","Data":"64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a"} Oct 29 22:03:19.707312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:19.707281 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:03:19.707467 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:19.707447 4952 patch_prober.go:29] interesting pod/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:03:19.707535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:19.707502 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:03:20.707814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:20.707781 4952 patch_prober.go:29] interesting pod/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:03:20.708221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:20.707835 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:03:21.266550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:21.266510 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:21.706952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:21.706921 4952 patch_prober.go:29] interesting pod/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:03:21.707118 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:21.706983 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:03:22.706922 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:22.706869 4952 patch_prober.go:29] interesting pod/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:03:22.707330 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:22.706966 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:03:23.079925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:23.079731 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:23.707027 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:23.706990 4952 patch_prober.go:29] interesting pod/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:03:23.707413 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:23.707044 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:03:24.711145 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:24.711115 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:03:24.711537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:24.711503 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:03:24.712120 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:24.712100 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:03:24.878986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:24.878934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:26.350869 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:26.350828 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:27.073456 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:27.073428 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:27.255953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:27.255919 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:27.432255 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:27.432221 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:27.738448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:27.738362 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:27.908180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:27.908139 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:29.249426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:29.249393 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:31.039092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:31.039050 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:32.787511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:32.787474 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:34.472666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:34.472608 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:34.978898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:34.978847 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:35.157073 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:35.157035 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:35.407996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:35.406919 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:36.302398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:36.302362 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:36.481057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:36.481028 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:36.853932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:36.849715 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:37.051429 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:37.051394 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:37.545161 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:37.545126 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:38.388386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:38.388341 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:40.202973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:40.202934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:40.432351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:40.432315 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:40.617070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:40.617033 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:40.807046 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:40.796431 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:40.885986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:40.878827 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9"] Oct 29 22:03:41.032898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:41.032745 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:41.212046 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:41.211946 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:41.418131 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:41.418095 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:41.494418 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:41.494324 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerName="istio-proxy" containerID="cri-o://5688e7a4cc481d34b4ea40f8118464cf661b602a53edb21604729a3b1066602e" gracePeriod=30 Oct 29 22:03:42.943390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:42.943353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:43.365187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:43.364979 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:03:44.142956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:44.142919 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:44.381930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:44.375788 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:45.266558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.266504 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:45.465930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.465230 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:45.536288 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:03:45.536198 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:03:45.668013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.667974 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv"] Oct 29 22:03:45.674199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.674174 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.690817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.690788 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8h8r\" (UniqueName: \"kubernetes.io/projected/1a9fa251-15d8-4089-9a97-1e290f122b60-kube-api-access-v8h8r\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.690990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.690835 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9fa251-15d8-4089-9a97-1e290f122b60-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.690990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.690877 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-metrics-certs\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.691225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.691183 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-maas-api-tls\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.691619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.691599 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-442dc2-serving-cert\"" Oct 29 22:03:45.718104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.718069 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv"] Oct 29 22:03:45.793984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.793845 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8h8r\" (UniqueName: \"kubernetes.io/projected/1a9fa251-15d8-4089-9a97-1e290f122b60-kube-api-access-v8h8r\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.793984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.793931 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9fa251-15d8-4089-9a97-1e290f122b60-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.794324 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.794301 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-metrics-certs\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.794449 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.794434 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-maas-api-tls\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.795334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.795311 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9fa251-15d8-4089-9a97-1e290f122b60-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.795483 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:03:45.795467 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-serving-cert: secret "maas-api-e2e-ipp-a-442dc2-serving-cert" not found Oct 29 22:03:45.795550 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:03:45.795537 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-maas-api-tls podName:1a9fa251-15d8-4089-9a97-1e290f122b60 nodeName:}" failed. No retries permitted until 2026-10-29 22:03:46.295518562 +0000 UTC m=+1291.770106958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-maas-api-tls") pod "maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" (UID: "1a9fa251-15d8-4089-9a97-1e290f122b60") : secret "maas-api-e2e-ipp-a-442dc2-serving-cert" not found Oct 29 22:03:45.800978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.800951 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-metrics-certs\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.820979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.820949 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8h8r\" (UniqueName: \"kubernetes.io/projected/1a9fa251-15d8-4089-9a97-1e290f122b60-kube-api-access-v8h8r\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:45.971937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:45.962582 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:46.159928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:46.144441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:46.314943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:46.314888 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-maas-api-tls\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:46.317959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:46.317925 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-maas-api-tls\") pod \"maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:46.591919 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:46.591871 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:46.630369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:46.630242 4952 generic.go:356] "Generic (PLEG): container finished" podID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerID="5688e7a4cc481d34b4ea40f8118464cf661b602a53edb21604729a3b1066602e" exitCode=0 Oct 29 22:03:46.630369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:46.630315 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" event={"ID":"7f8b8b41-f299-4b37-83f5-82bec45f7517","Type":"ContainerDied","Data":"5688e7a4cc481d34b4ea40f8118464cf661b602a53edb21604729a3b1066602e"} Oct 29 22:03:46.954720 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:46.954677 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv"] Oct 29 22:03:47.050031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.050006 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 22:03:47.298155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.297441 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8b8b41-f299-4b37-83f5-82bec45f7517-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8b8b41-f299-4b37-83f5-82bec45f7517-istiod-ca-cert\") pod \"7f8b8b41-f299-4b37-83f5-82bec45f7517\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " Oct 29 22:03:47.298155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.297577 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-kube-api-access-lqgvg\" (UniqueName: \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-kube-api-access-lqgvg\") pod \"7f8b8b41-f299-4b37-83f5-82bec45f7517\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " Oct 29 22:03:47.298155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.297701 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-data\") pod \"7f8b8b41-f299-4b37-83f5-82bec45f7517\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " Oct 29 22:03:47.298155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.297783 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-envoy\") pod \"7f8b8b41-f299-4b37-83f5-82bec45f7517\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " Oct 29 22:03:47.298155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.297811 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-token\" (UniqueName: \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-token\") pod \"7f8b8b41-f299-4b37-83f5-82bec45f7517\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " Oct 29 22:03:47.298155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.297862 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-socket\") pod \"7f8b8b41-f299-4b37-83f5-82bec45f7517\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " Oct 29 22:03:47.298155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.297984 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-credential-socket\") pod \"7f8b8b41-f299-4b37-83f5-82bec45f7517\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " Oct 29 22:03:47.298155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.298017 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-podinfo\") pod \"7f8b8b41-f299-4b37-83f5-82bec45f7517\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " Oct 29 22:03:47.298155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.298054 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-certs\") pod \"7f8b8b41-f299-4b37-83f5-82bec45f7517\" (UID: \"7f8b8b41-f299-4b37-83f5-82bec45f7517\") " Oct 29 22:03:47.298820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.298261 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-data" pod "7f8b8b41-f299-4b37-83f5-82bec45f7517" (UID: "7f8b8b41-f299-4b37-83f5-82bec45f7517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:47.298820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.298748 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:03:47.299003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.298980 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-socket" pod "7f8b8b41-f299-4b37-83f5-82bec45f7517" (UID: "7f8b8b41-f299-4b37-83f5-82bec45f7517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:47.299459 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.299435 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-credential-socket" pod "7f8b8b41-f299-4b37-83f5-82bec45f7517" (UID: "7f8b8b41-f299-4b37-83f5-82bec45f7517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:47.299797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.299733 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8b8b41-f299-4b37-83f5-82bec45f7517-istiod-ca-cert" pod "7f8b8b41-f299-4b37-83f5-82bec45f7517" (UID: "7f8b8b41-f299-4b37-83f5-82bec45f7517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:03:47.302475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.302447 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-envoy" pod "7f8b8b41-f299-4b37-83f5-82bec45f7517" (UID: "7f8b8b41-f299-4b37-83f5-82bec45f7517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:47.302736 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.302700 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-certs" pod "7f8b8b41-f299-4b37-83f5-82bec45f7517" (UID: "7f8b8b41-f299-4b37-83f5-82bec45f7517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:03:47.305377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.305345 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-kube-api-access-lqgvg" pod "7f8b8b41-f299-4b37-83f5-82bec45f7517" (UID: "7f8b8b41-f299-4b37-83f5-82bec45f7517"). InnerVolumeSpecName "kube-api-access-lqgvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:47.308443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.308422 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-podinfo" pod "7f8b8b41-f299-4b37-83f5-82bec45f7517" (UID: "7f8b8b41-f299-4b37-83f5-82bec45f7517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:03:47.309066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.309033 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-token" pod "7f8b8b41-f299-4b37-83f5-82bec45f7517" (UID: "7f8b8b41-f299-4b37-83f5-82bec45f7517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:03:47.401807 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.401763 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:03:47.401807 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.401807 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:03:47.402460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.401825 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8b8b41-f299-4b37-83f5-82bec45f7517-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:03:47.402460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.401837 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqgvg\" (UniqueName: \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-kube-api-access-lqgvg\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:03:47.402460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.401851 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:03:47.402460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.401864 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8b8b41-f299-4b37-83f5-82bec45f7517-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:03:47.402460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.401875 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:03:47.402460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.401889 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8b8b41-f299-4b37-83f5-82bec45f7517-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:03:47.659718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.659677 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" event={"ID":"7f8b8b41-f299-4b37-83f5-82bec45f7517","Type":"ContainerDied","Data":"3a1f207e4b365828a5ff24f6d6c4bf05c3ed52239c994cd788fbf9998547fc0e"} Oct 29 22:03:47.659958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.659750 4952 scope.go:122] "RemoveContainer" containerID="5688e7a4cc481d34b4ea40f8118464cf661b602a53edb21604729a3b1066602e" Oct 29 22:03:47.659958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.659928 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9" Oct 29 22:03:47.678051 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.672394 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" event={"ID":"1a9fa251-15d8-4089-9a97-1e290f122b60","Type":"ContainerStarted","Data":"0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4"} Oct 29 22:03:47.678051 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.672455 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" event={"ID":"1a9fa251-15d8-4089-9a97-1e290f122b60","Type":"ContainerStarted","Data":"e9bffe3887b52dfd8cee7fc4f60707b0ecd8f2fe66428e70b08b9ee256111e5b"} Oct 29 22:03:47.678051 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.673317 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:47.702678 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.702650 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:47.734028 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.733966 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" podStartSLOduration=2.259684267 podStartE2EDuration="2.733948695s" totalImagesPullingTime="474.264428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:47.732614444 +0000 UTC m=+1293.207202865" watchObservedRunningTime="2026-10-29 22:03:47.733948695 +0000 UTC m=+1293.208537116" Oct 29 22:03:47.777214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.777162 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9"] Oct 29 22:03:47.783817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.783785 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-wxjn9"] Oct 29 22:03:47.904536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:47.904448 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:48.184356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:48.184310 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:48.650812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:48.650781 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" path="/var/lib/kubelet/pods/7f8b8b41-f299-4b37-83f5-82bec45f7517/volumes" Oct 29 22:03:49.944535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:49.944501 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:51.449122 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:51.449085 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:51.619702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:51.619631 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:51.950245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:51.950214 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:53.520753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.520717 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7776-1e3e13"] Oct 29 22:03:53.521436 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.521421 4952 cpu_manager.go:414] "containerMap: removing container" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerName="istio-proxy" Oct 29 22:03:53.521477 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.521438 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerName="istio-proxy" Oct 29 22:03:53.521568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.521560 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8b8b41-f299-4b37-83f5-82bec45f7517" containerName="istio-proxy" Oct 29 22:03:53.526651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.526632 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1e3e13" Oct 29 22:03:53.545237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.545205 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7776-1e3e13"] Oct 29 22:03:53.547883 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.547859 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r84ws\"" Oct 29 22:03:53.578749 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.578714 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb62b\" (UniqueName: \"kubernetes.io/projected/6cef4628-2ce9-4820-86e2-1fad0e592859-kube-api-access-hb62b\") pod \"test-curl-7776-1e3e13\" (UID: \"6cef4628-2ce9-4820-86e2-1fad0e592859\") " pod="opendatahub/test-curl-7776-1e3e13" Oct 29 22:03:53.680836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.680800 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb62b\" (UniqueName: \"kubernetes.io/projected/6cef4628-2ce9-4820-86e2-1fad0e592859-kube-api-access-hb62b\") pod \"test-curl-7776-1e3e13\" (UID: \"6cef4628-2ce9-4820-86e2-1fad0e592859\") " pod="opendatahub/test-curl-7776-1e3e13" Oct 29 22:03:53.705247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.705214 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb62b\" (UniqueName: \"kubernetes.io/projected/6cef4628-2ce9-4820-86e2-1fad0e592859-kube-api-access-hb62b\") pod \"test-curl-7776-1e3e13\" (UID: \"6cef4628-2ce9-4820-86e2-1fad0e592859\") " pod="opendatahub/test-curl-7776-1e3e13" Oct 29 22:03:53.848377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:53.848339 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1e3e13" Oct 29 22:03:54.170036 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:54.170005 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7776-1e3e13"] Oct 29 22:03:54.170724 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:03:54.170699 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cef4628_2ce9_4820_86e2_1fad0e592859.slice/crio-433198449adb05b74f117246db7e6c271a144dd23e56e18de2911e2281071d3f WatchSource:0}: Error finding container 433198449adb05b74f117246db7e6c271a144dd23e56e18de2911e2281071d3f: Status 404 returned error can't find the container with id 433198449adb05b74f117246db7e6c271a144dd23e56e18de2911e2281071d3f Oct 29 22:03:54.238208 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:54.238172 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:54.705526 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:54.705499 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:03:54.741569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:54.741530 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7776-1e3e13" event={"ID":"6cef4628-2ce9-4820-86e2-1fad0e592859","Type":"ContainerStarted","Data":"433198449adb05b74f117246db7e6c271a144dd23e56e18de2911e2281071d3f"} Oct 29 22:03:55.756792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:55.756753 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7776-1e3e13" event={"ID":"6cef4628-2ce9-4820-86e2-1fad0e592859","Type":"ContainerStarted","Data":"22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a"} Oct 29 22:03:55.800245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:55.800190 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7776-1e3e13" podStartSLOduration=2.082124448 podStartE2EDuration="2.800174241s" totalImagesPullingTime="718.049793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:55.796002892 +0000 UTC m=+1301.270591311" watchObservedRunningTime="2026-10-29 22:03:55.800174241 +0000 UTC m=+1301.274762659" Oct 29 22:03:55.905749 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:55.905657 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:56.685580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:56.685546 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7776-1e3e13"] Oct 29 22:03:56.688293 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:56.688223 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7776-1e3e13"] Oct 29 22:03:57.772366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:57.772283 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7776-1e3e13" podUID="6cef4628-2ce9-4820-86e2-1fad0e592859" containerName="test-curl-7776-1e3e13" containerID="cri-o://22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a" gracePeriod=2 Oct 29 22:03:57.785673 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:03:57.785642 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7776-1e3e13\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="6cef4628-2ce9-4820-86e2-1fad0e592859" pod="opendatahub/test-curl-7776-1e3e13" Oct 29 22:03:57.983949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:57.983159 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:59.318353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:59.318316 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:03:59.502327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:03:59.502286 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:00.019129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.019103 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1e3e13" Oct 29 22:04:00.141327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.141285 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cef4628-2ce9-4820-86e2-1fad0e592859-kube-api-access-hb62b\" (UniqueName: \"kubernetes.io/projected/6cef4628-2ce9-4820-86e2-1fad0e592859-kube-api-access-hb62b\") pod \"6cef4628-2ce9-4820-86e2-1fad0e592859\" (UID: \"6cef4628-2ce9-4820-86e2-1fad0e592859\") " Oct 29 22:04:00.144002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.143973 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cef4628-2ce9-4820-86e2-1fad0e592859-kube-api-access-hb62b" pod "6cef4628-2ce9-4820-86e2-1fad0e592859" (UID: "6cef4628-2ce9-4820-86e2-1fad0e592859"). InnerVolumeSpecName "kube-api-access-hb62b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:00.244996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.244958 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb62b\" (UniqueName: \"kubernetes.io/projected/6cef4628-2ce9-4820-86e2-1fad0e592859-kube-api-access-hb62b\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:00.488998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.488863 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:00.646761 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.646726 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cef4628-2ce9-4820-86e2-1fad0e592859" path="/var/lib/kubelet/pods/6cef4628-2ce9-4820-86e2-1fad0e592859/volumes" Oct 29 22:04:00.809408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.809315 4952 generic.go:356] "Generic (PLEG): container finished" podID="6cef4628-2ce9-4820-86e2-1fad0e592859" containerID="22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a" exitCode=137 Oct 29 22:04:00.809408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.809371 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1e3e13" Oct 29 22:04:00.809408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.809387 4952 scope.go:122] "RemoveContainer" containerID="22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a" Oct 29 22:04:00.819440 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.819422 4952 scope.go:122] "RemoveContainer" containerID="22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a" Oct 29 22:04:00.819731 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:04:00.819709 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a\": container with ID starting with 22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a not found: ID does not exist" containerID="22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a" Oct 29 22:04:00.819795 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:00.819744 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a"} err="failed to get container status \"22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a\": rpc error: code = NotFound desc = could not find container \"22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a\": container with ID starting with 22ef5b331c4e9d14d541c327a30c8ea33bbcb59be17ec283775918c93613aa4a not found: ID does not exist" Oct 29 22:04:02.231801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:02.231768 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:03.934867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:03.934830 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:05.631812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:05.631774 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:07.341272 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:07.341237 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:07.714812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:07.714776 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:07.895375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:07.895341 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:09.146608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:09.146568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:10.728121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.728088 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g"] Oct 29 22:04:10.728997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.728974 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6cef4628-2ce9-4820-86e2-1fad0e592859" containerName="test-curl-7776-1e3e13" Oct 29 22:04:10.728997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.728996 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cef4628-2ce9-4820-86e2-1fad0e592859" containerName="test-curl-7776-1e3e13" Oct 29 22:04:10.729196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.729182 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cef4628-2ce9-4820-86e2-1fad0e592859" containerName="test-curl-7776-1e3e13" Oct 29 22:04:10.732919 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.732887 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.760442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.760408 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3fe45b-serving-cert\"" Oct 29 22:04:10.767679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.767632 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g"] Oct 29 22:04:10.832124 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.832088 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.832281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.832144 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.832340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.832312 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmw8\" (UniqueName: \"kubernetes.io/projected/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-kube-api-access-dvmw8\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.832550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.832532 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.896130 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.896098 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:10.934541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.934511 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.935127 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.934627 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.935127 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.934680 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.935127 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.934770 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmw8\" (UniqueName: \"kubernetes.io/projected/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-kube-api-access-dvmw8\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.935436 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.935411 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.938382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.938356 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.938556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.938454 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:10.977493 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:10.977400 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmw8\" (UniqueName: \"kubernetes.io/projected/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-kube-api-access-dvmw8\") pod \"maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:11.049988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:11.049858 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:11.073983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:11.073956 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:11.396614 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:04:11.396570 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7008d6_5941_4e8b_b0d3_a576a186c8d0.slice/crio-003bb297fc2d089645d38905a0e06d32cb585a834e9c41fccf253b7fa1633ba0 WatchSource:0}: Error finding container 003bb297fc2d089645d38905a0e06d32cb585a834e9c41fccf253b7fa1633ba0: Status 404 returned error can't find the container with id 003bb297fc2d089645d38905a0e06d32cb585a834e9c41fccf253b7fa1633ba0 Oct 29 22:04:11.396869 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:11.396847 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g"] Oct 29 22:04:11.944947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:11.940963 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" event={"ID":"8a7008d6-5941-4e8b-b0d3-a576a186c8d0","Type":"ContainerStarted","Data":"003bb297fc2d089645d38905a0e06d32cb585a834e9c41fccf253b7fa1633ba0"} Oct 29 22:04:12.739002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:12.738965 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:12.964927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:12.964136 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" event={"ID":"8a7008d6-5941-4e8b-b0d3-a576a186c8d0","Type":"ContainerStarted","Data":"5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc"} Oct 29 22:04:12.964927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:12.964408 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:13.054441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:13.054342 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" podStartSLOduration=2.635298595 podStartE2EDuration="3.054328958s" totalImagesPullingTime="419.030363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:13.051209191 +0000 UTC m=+1318.525797610" watchObservedRunningTime="2026-10-29 22:04:13.054328958 +0000 UTC m=+1318.528917377" Oct 29 22:04:14.686689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:14.686654 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:15.528612 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:04:15.528543 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:04:16.836591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:16.836550 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z"] Oct 29 22:04:16.844305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:16.844272 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:16.866530 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:16.866487 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z"] Oct 29 22:04:16.870852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:16.870824 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4d79fa-serving-cert\"" Oct 29 22:04:16.951348 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:16.951312 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:16.951563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:16.951362 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51468c4a-574a-488c-9b46-0876a15df246-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:16.951563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:16.951469 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpjf\" (UniqueName: \"kubernetes.io/projected/51468c4a-574a-488c-9b46-0876a15df246-kube-api-access-6xpjf\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:16.951714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:16.951686 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-metrics-certs\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:16.990994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:16.990947 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:17.055256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.054553 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:17.055256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.054654 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51468c4a-574a-488c-9b46-0876a15df246-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:17.055256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.054743 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpjf\" (UniqueName: \"kubernetes.io/projected/51468c4a-574a-488c-9b46-0876a15df246-kube-api-access-6xpjf\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:17.055256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.055010 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-metrics-certs\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:17.060035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.056595 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51468c4a-574a-488c-9b46-0876a15df246-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:17.060035 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:04:17.056712 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-serving-cert: secret "maas-api-e2e-praxis-a-4d79fa-serving-cert" not found Oct 29 22:04:17.060035 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:04:17.056787 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-maas-api-tls podName:51468c4a-574a-488c-9b46-0876a15df246 nodeName:}" failed. No retries permitted until 2026-10-29 22:04:17.556767151 +0000 UTC m=+1323.031355553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-maas-api-tls") pod "maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" (UID: "51468c4a-574a-488c-9b46-0876a15df246") : secret "maas-api-e2e-praxis-a-4d79fa-serving-cert" not found Oct 29 22:04:17.094573 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.094484 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpjf\" (UniqueName: \"kubernetes.io/projected/51468c4a-574a-488c-9b46-0876a15df246-kube-api-access-6xpjf\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:17.104455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.104410 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-metrics-certs\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:17.231993 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.231953 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:17.411362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.411265 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:17.571549 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.564954 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:17.592925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.585424 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:17.607262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.607225 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:17.759892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:17.759799 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:18.132739 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:04:18.132670 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51468c4a_574a_488c_9b46_0876a15df246.slice/crio-103ce889af4244a8aa965766537994069ec31b79ca6a2db4e0d34646b3139989 WatchSource:0}: Error finding container 103ce889af4244a8aa965766537994069ec31b79ca6a2db4e0d34646b3139989: Status 404 returned error can't find the container with id 103ce889af4244a8aa965766537994069ec31b79ca6a2db4e0d34646b3139989 Oct 29 22:04:18.134633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:18.134609 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:04:18.135140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:18.135112 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z"] Oct 29 22:04:18.509829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:18.509743 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:18.678754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:18.678718 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:18.978550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:18.978517 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:04:19.027253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:19.027218 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" event={"ID":"51468c4a-574a-488c-9b46-0876a15df246","Type":"ContainerStarted","Data":"3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc"} Oct 29 22:04:19.027426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:19.027281 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" event={"ID":"51468c4a-574a-488c-9b46-0876a15df246","Type":"ContainerStarted","Data":"103ce889af4244a8aa965766537994069ec31b79ca6a2db4e0d34646b3139989"} Oct 29 22:04:19.028075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:19.028053 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:19.106723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:19.106652 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" podStartSLOduration=2.679197963 podStartE2EDuration="3.106630354s" totalImagesPullingTime="427.432391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:19.099182507 +0000 UTC m=+1324.573770925" watchObservedRunningTime="2026-10-29 22:04:19.106630354 +0000 UTC m=+1324.581218774" Oct 29 22:04:19.223302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:19.223248 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:19.517925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:19.517535 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:04:21.128311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:21.128275 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:22.148881 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:22.148842 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:22.435973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:22.435466 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:22.792263 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:22.792159 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:24.308406 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:24.304221 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:25.213752 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:25.213714 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:25.382857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:25.382825 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:26.172499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:26.172468 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:04:27.068988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:27.068951 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:27.883523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:27.883486 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:29.022230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:29.013766 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:29.115097 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:29.115059 4952 patch_prober.go:29] interesting pod/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:04:29.115300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:29.115133 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:04:30.737651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:30.737613 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:31.310968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:31.310890 4952 generic.go:356] "Generic (PLEG): container finished" podID="184e1774-f710-4a0c-a816-af20d9032d86" containerID="4d532fb60a6fa0b8433c1b99a39cd7f4efe3de1df385498cd93ec4eff11ff377" exitCode=0 Oct 29 22:04:31.311197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:31.310972 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" event={"ID":"184e1774-f710-4a0c-a816-af20d9032d86","Type":"ContainerDied","Data":"4d532fb60a6fa0b8433c1b99a39cd7f4efe3de1df385498cd93ec4eff11ff377"} Oct 29 22:04:31.311436 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:31.311422 4952 scope.go:122] "RemoveContainer" containerID="4d532fb60a6fa0b8433c1b99a39cd7f4efe3de1df385498cd93ec4eff11ff377" Oct 29 22:04:31.313168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:31.313137 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:04:31.313294 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:31.313224 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:04:31.313294 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:31.313272 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:04:32.320389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:32.320354 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" event={"ID":"184e1774-f710-4a0c-a816-af20d9032d86","Type":"ContainerStarted","Data":"710abeab757b8e349a2a565f0850be918338fa0b2ebd2c4237fe83c59163144b"} Oct 29 22:04:33.008093 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:33.008033 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:33.178116 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:33.178077 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9"] Oct 29 22:04:33.178383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:33.178323 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" containerID="cri-o://bd13e8df9c05bd41165795ecbdbdcf2d4c687ad5d840c498d2bd6a4a9838537f" gracePeriod=30 Oct 29 22:04:33.203359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:33.203324 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:04:33.203681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:33.203651 4952 patch_prober.go:29] interesting pod/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:04:33.203798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:33.203709 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 29 22:04:34.203680 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:34.203647 4952 patch_prober.go:29] interesting pod/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:04:34.204131 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:34.203717 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 29 22:04:34.800072 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:34.799993 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:35.203091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:35.203056 4952 patch_prober.go:29] interesting pod/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:04:35.203271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:35.203124 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 29 22:04:36.202870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:36.202825 4952 patch_prober.go:29] interesting pod/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:04:36.203423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:36.202922 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 29 22:04:36.702243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:36.702205 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:37.203363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:37.203322 4952 patch_prober.go:29] interesting pod/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:04:37.203979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:37.203941 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 29 22:04:38.204266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:38.204226 4952 patch_prober.go:29] interesting pod/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 29 22:04:38.204726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:38.204303 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 29 22:04:38.253651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:38.253556 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:38.476973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:38.471054 4952 generic.go:356] "Generic (PLEG): container finished" podID="35194ac9-c066-4437-b1b1-35defc3e122c" containerID="bd13e8df9c05bd41165795ecbdbdcf2d4c687ad5d840c498d2bd6a4a9838537f" exitCode=0 Oct 29 22:04:38.476973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:38.471190 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" event={"ID":"35194ac9-c066-4437-b1b1-35defc3e122c","Type":"ContainerDied","Data":"bd13e8df9c05bd41165795ecbdbdcf2d4c687ad5d840c498d2bd6a4a9838537f"} Oct 29 22:04:39.036630 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.036601 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 22:04:39.071036 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.071003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:39.125350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.125312 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-envoy\") pod \"35194ac9-c066-4437-b1b1-35defc3e122c\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " Oct 29 22:04:39.125643 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.125615 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-socket\") pod \"35194ac9-c066-4437-b1b1-35defc3e122c\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " Oct 29 22:04:39.125795 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.125700 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-credential-socket\") pod \"35194ac9-c066-4437-b1b1-35defc3e122c\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " Oct 29 22:04:39.125795 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.125783 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-data\") pod \"35194ac9-c066-4437-b1b1-35defc3e122c\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " Oct 29 22:04:39.125927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.125830 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-kube-api-access-mgt5k\" (UniqueName: \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-kube-api-access-mgt5k\") pod \"35194ac9-c066-4437-b1b1-35defc3e122c\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " Oct 29 22:04:39.125927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.125889 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-certs\") pod \"35194ac9-c066-4437-b1b1-35defc3e122c\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " Oct 29 22:04:39.126040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.125978 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35194ac9-c066-4437-b1b1-35defc3e122c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35194ac9-c066-4437-b1b1-35defc3e122c-istiod-ca-cert\") pod \"35194ac9-c066-4437-b1b1-35defc3e122c\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " Oct 29 22:04:39.126089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.126075 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-istio-token\" (UniqueName: \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-istio-token\") pod \"35194ac9-c066-4437-b1b1-35defc3e122c\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " Oct 29 22:04:39.126183 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.126165 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35194ac9-c066-4437-b1b1-35defc3e122c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35194ac9-c066-4437-b1b1-35defc3e122c-istio-podinfo\") pod \"35194ac9-c066-4437-b1b1-35defc3e122c\" (UID: \"35194ac9-c066-4437-b1b1-35defc3e122c\") " Oct 29 22:04:39.126251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.126200 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-socket" pod "35194ac9-c066-4437-b1b1-35defc3e122c" (UID: "35194ac9-c066-4437-b1b1-35defc3e122c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:39.126679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.126655 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-credential-socket" pod "35194ac9-c066-4437-b1b1-35defc3e122c" (UID: "35194ac9-c066-4437-b1b1-35defc3e122c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:39.126886 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.126865 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:39.127085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.127046 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-data" pod "35194ac9-c066-4437-b1b1-35defc3e122c" (UID: "35194ac9-c066-4437-b1b1-35defc3e122c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:39.127189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.127099 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-certs" pod "35194ac9-c066-4437-b1b1-35defc3e122c" (UID: "35194ac9-c066-4437-b1b1-35defc3e122c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:39.127571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.127531 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35194ac9-c066-4437-b1b1-35defc3e122c-istiod-ca-cert" pod "35194ac9-c066-4437-b1b1-35defc3e122c" (UID: "35194ac9-c066-4437-b1b1-35defc3e122c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:39.134798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.134283 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-kube-api-access-mgt5k" pod "35194ac9-c066-4437-b1b1-35defc3e122c" (UID: "35194ac9-c066-4437-b1b1-35defc3e122c"). InnerVolumeSpecName "kube-api-access-mgt5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:39.134798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.134372 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-envoy" pod "35194ac9-c066-4437-b1b1-35defc3e122c" (UID: "35194ac9-c066-4437-b1b1-35defc3e122c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:39.136242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.135365 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-istio-token" pod "35194ac9-c066-4437-b1b1-35defc3e122c" (UID: "35194ac9-c066-4437-b1b1-35defc3e122c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:39.136242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.136160 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35194ac9-c066-4437-b1b1-35defc3e122c-istio-podinfo" pod "35194ac9-c066-4437-b1b1-35defc3e122c" (UID: "35194ac9-c066-4437-b1b1-35defc3e122c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:04:39.208506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.208472 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:04:39.208964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.208924 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:04:39.209799 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.209779 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:04:39.229010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.228975 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgt5k\" (UniqueName: \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-kube-api-access-mgt5k\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:39.229010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.229014 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:39.229229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.229031 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35194ac9-c066-4437-b1b1-35defc3e122c-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:39.229229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.229045 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35194ac9-c066-4437-b1b1-35defc3e122c-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:39.229229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.229057 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35194ac9-c066-4437-b1b1-35defc3e122c-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:39.229229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.229070 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:39.229229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.229083 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:39.229229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.229098 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35194ac9-c066-4437-b1b1-35defc3e122c-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:39.236710 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.236670 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:39.415082 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.415045 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:39.494518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.494489 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" Oct 29 22:04:39.494518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.494509 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9" event={"ID":"35194ac9-c066-4437-b1b1-35defc3e122c","Type":"ContainerDied","Data":"db04381aa88064add7f60b997753bb7bf122552184477ef2da57f02c21203a99"} Oct 29 22:04:39.494828 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.494588 4952 scope.go:122] "RemoveContainer" containerID="bd13e8df9c05bd41165795ecbdbdcf2d4c687ad5d840c498d2bd6a4a9838537f" Oct 29 22:04:39.560457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.560421 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9"] Oct 29 22:04:39.564419 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.564394 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-8rrz9"] Oct 29 22:04:39.765590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:39.765488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:40.648556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:40.648512 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" path="/var/lib/kubelet/pods/35194ac9-c066-4437-b1b1-35defc3e122c/volumes" Oct 29 22:04:41.167939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:41.162752 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:42.947998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:42.946306 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:44.720169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:44.720132 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:45.553688 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:04:45.553637 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:04:46.699500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:46.699462 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:46.873529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:46.873492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:47.086112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:47.086075 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns"] Oct 29 22:04:47.086369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:47.086338 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" containerID="cri-o://64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a" gracePeriod=30 Oct 29 22:04:47.195449 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:47.195407 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:47.683492 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:47.683462 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:47.859519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:47.859480 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:48.111447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:48.111407 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:48.293987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:48.293950 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:49.149336 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:49.149294 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:49.328069 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:49.328034 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:49.526571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:49.526480 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:49.745777 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:49.745738 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:49.925285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:49.925233 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:50.096045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:50.096004 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:50.445443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:50.445401 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:51.993401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:51.993363 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:52.397960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.397930 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:04:52.523459 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.523368 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-podinfo\") pod \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " Oct 29 22:04:52.523459 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.523434 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istiod-ca-cert\") pod \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " Oct 29 22:04:52.524060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.523514 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-kube-api-access-jq6s7\" (UniqueName: \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-kube-api-access-jq6s7\") pod \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " Oct 29 22:04:52.524060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.523562 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-credential-socket\") pod \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " Oct 29 22:04:52.524060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.523773 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-token\" (UniqueName: \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-token\") pod \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " Oct 29 22:04:52.524060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.523806 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istiod-ca-cert" pod "925f097f-4d45-4843-b5f6-ef6412b3dfa9" (UID: "925f097f-4d45-4843-b5f6-ef6412b3dfa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:52.524295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.524255 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-data\") pod \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " Oct 29 22:04:52.524628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.524581 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-data" pod "925f097f-4d45-4843-b5f6-ef6412b3dfa9" (UID: "925f097f-4d45-4843-b5f6-ef6412b3dfa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:52.524816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.524797 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-certs\") pod \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " Oct 29 22:04:52.525012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.524880 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-certs" pod "925f097f-4d45-4843-b5f6-ef6412b3dfa9" (UID: "925f097f-4d45-4843-b5f6-ef6412b3dfa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:52.525012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.524884 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-credential-socket" pod "925f097f-4d45-4843-b5f6-ef6412b3dfa9" (UID: "925f097f-4d45-4843-b5f6-ef6412b3dfa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:52.525251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.524978 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-socket\") pod \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " Oct 29 22:04:52.525251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.525067 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-envoy\") pod \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\" (UID: \"925f097f-4d45-4843-b5f6-ef6412b3dfa9\") " Oct 29 22:04:52.525251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.525179 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-socket" pod "925f097f-4d45-4843-b5f6-ef6412b3dfa9" (UID: "925f097f-4d45-4843-b5f6-ef6412b3dfa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:52.526030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.525988 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.526030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.526015 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.526253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.526030 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.526253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.526046 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.526253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.526059 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.526435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.526408 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-kube-api-access-jq6s7" pod "925f097f-4d45-4843-b5f6-ef6412b3dfa9" (UID: "925f097f-4d45-4843-b5f6-ef6412b3dfa9"). InnerVolumeSpecName "kube-api-access-jq6s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:52.526700 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.526678 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-token" pod "925f097f-4d45-4843-b5f6-ef6412b3dfa9" (UID: "925f097f-4d45-4843-b5f6-ef6412b3dfa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:52.526916 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.526886 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-podinfo" pod "925f097f-4d45-4843-b5f6-ef6412b3dfa9" (UID: "925f097f-4d45-4843-b5f6-ef6412b3dfa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:04:52.527709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.527680 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-envoy" pod "925f097f-4d45-4843-b5f6-ef6412b3dfa9" (UID: "925f097f-4d45-4843-b5f6-ef6412b3dfa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:04:52.627441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.627405 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.627441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.627440 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.627658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.627451 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq6s7\" (UniqueName: \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-kube-api-access-jq6s7\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.627658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.627460 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925f097f-4d45-4843-b5f6-ef6412b3dfa9-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:52.665553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.660182 4952 generic.go:356] "Generic (PLEG): container finished" podID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerID="64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a" exitCode=0 Oct 29 22:04:52.665553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.660660 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" Oct 29 22:04:52.666996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.666956 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" event={"ID":"925f097f-4d45-4843-b5f6-ef6412b3dfa9","Type":"ContainerDied","Data":"64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a"} Oct 29 22:04:52.667147 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.667033 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns" event={"ID":"925f097f-4d45-4843-b5f6-ef6412b3dfa9","Type":"ContainerDied","Data":"1746eb05bbfd51eccd839a3d0a450f069ee1b6d7a6b43a0ddf4e84ccc8f6dabb"} Oct 29 22:04:52.667147 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.667076 4952 scope.go:122] "RemoveContainer" containerID="64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a" Oct 29 22:04:52.683999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.683973 4952 scope.go:122] "RemoveContainer" containerID="009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5" Oct 29 22:04:52.697271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.697243 4952 scope.go:122] "RemoveContainer" containerID="64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a" Oct 29 22:04:52.697647 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:04:52.697616 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a\": container with ID starting with 64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a not found: ID does not exist" containerID="64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a" Oct 29 22:04:52.697744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.697658 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a"} err="failed to get container status \"64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a\": rpc error: code = NotFound desc = could not find container \"64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a\": container with ID starting with 64b90add232cbd80942c5489c1f7dfa13cf2c51fe5dd7622bf0e0f319335662a not found: ID does not exist" Oct 29 22:04:52.697744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.697684 4952 scope.go:122] "RemoveContainer" containerID="009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5" Oct 29 22:04:52.698009 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:04:52.697989 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5\": container with ID starting with 009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5 not found: ID does not exist" containerID="009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5" Oct 29 22:04:52.698086 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.698017 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5"} err="failed to get container status \"009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5\": rpc error: code = NotFound desc = could not find container \"009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5\": container with ID starting with 009e4d3f82858d2742c09c61487d0a60230054bd1ccd51c4bcbc819c0b7342a5 not found: ID does not exist" Oct 29 22:04:52.727001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.726930 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns"] Oct 29 22:04:52.732034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:52.732007 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fcce6b-openshift-default-5c756d8f55-4hpns"] Oct 29 22:04:53.195774 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:53.195732 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:53.404931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:53.404467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:53.578014 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:53.576983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:54.533846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:54.533797 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:54.649375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:54.649339 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" path="/var/lib/kubelet/pods/925f097f-4d45-4843-b5f6-ef6412b3dfa9/volumes" Oct 29 22:04:55.967809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:55.964686 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h"] Oct 29 22:04:55.969989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:55.964950 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" podUID="594282a4-47b3-425b-8f9e-1ffac4f2cdc6" containerName="maas-api" containerID="cri-o://a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b" gracePeriod=30 Oct 29 22:04:56.070305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.070271 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:56.145738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.145700 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv"] Oct 29 22:04:56.146294 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.146263 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" podUID="b6ad5968-00d3-4048-a03d-7e5168abd368" containerName="payload-pre-processing" containerID="cri-o://3a74a8fc58196b948e4c8bbdd75ed94ab8faf8f04b86ce6df3d3634d8cdb9588" gracePeriod=30 Oct 29 22:04:56.253464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.253371 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:56.403804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.403777 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:04:56.423413 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.423371 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-maas-api-tls\") pod \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " Oct 29 22:04:56.425814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.425771 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:56.427314 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.427285 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-maas-api-tls" pod "594282a4-47b3-425b-8f9e-1ffac4f2cdc6" (UID: "594282a4-47b3-425b-8f9e-1ffac4f2cdc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:56.437061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.432030 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-openshift-service-ca\") pod \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " Oct 29 22:04:56.437061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.432177 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-metrics-certs\") pod \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " Oct 29 22:04:56.437061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.432294 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-kube-api-access-n9xbh\" (UniqueName: \"kubernetes.io/projected/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-kube-api-access-n9xbh\") pod \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\" (UID: \"594282a4-47b3-425b-8f9e-1ffac4f2cdc6\") " Oct 29 22:04:56.437061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.433331 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-openshift-service-ca" pod "594282a4-47b3-425b-8f9e-1ffac4f2cdc6" (UID: "594282a4-47b3-425b-8f9e-1ffac4f2cdc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:56.459855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.448066 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:56.460685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.460654 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:56.474886 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.474828 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-metrics-certs" pod "594282a4-47b3-425b-8f9e-1ffac4f2cdc6" (UID: "594282a4-47b3-425b-8f9e-1ffac4f2cdc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:04:56.479555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.475268 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-kube-api-access-n9xbh" pod "594282a4-47b3-425b-8f9e-1ffac4f2cdc6" (UID: "594282a4-47b3-425b-8f9e-1ffac4f2cdc6"). InnerVolumeSpecName "kube-api-access-n9xbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:56.719104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.719050 4952 generic.go:356] "Generic (PLEG): container finished" podID="594282a4-47b3-425b-8f9e-1ffac4f2cdc6" containerID="a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b" exitCode=0 Oct 29 22:04:56.719608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.719581 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" Oct 29 22:04:56.722770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.722743 4952 generic.go:356] "Generic (PLEG): container finished" podID="b6ad5968-00d3-4048-a03d-7e5168abd368" containerID="3a74a8fc58196b948e4c8bbdd75ed94ab8faf8f04b86ce6df3d3634d8cdb9588" exitCode=0 Oct 29 22:04:56.784751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.784722 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:56.786970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.785779 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9xbh\" (UniqueName: \"kubernetes.io/projected/594282a4-47b3-425b-8f9e-1ffac4f2cdc6-kube-api-access-n9xbh\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:56.815712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.815676 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" event={"ID":"594282a4-47b3-425b-8f9e-1ffac4f2cdc6","Type":"ContainerDied","Data":"a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b"} Oct 29 22:04:56.816035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.816016 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h" event={"ID":"594282a4-47b3-425b-8f9e-1ffac4f2cdc6","Type":"ContainerDied","Data":"140ad8505466281d3830916130ae68755de1635e4aabe0169854f9af4230ec8a"} Oct 29 22:04:56.816164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.816152 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" event={"ID":"b6ad5968-00d3-4048-a03d-7e5168abd368","Type":"ContainerDied","Data":"3a74a8fc58196b948e4c8bbdd75ed94ab8faf8f04b86ce6df3d3634d8cdb9588"} Oct 29 22:04:56.817026 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.816084 4952 scope.go:122] "RemoveContainer" containerID="a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b" Oct 29 22:04:56.838379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.838328 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:56.839453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.839414 4952 scope.go:122] "RemoveContainer" containerID="a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b" Oct 29 22:04:56.839965 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:04:56.839833 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b\": container with ID starting with a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b not found: ID does not exist" containerID="a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b" Oct 29 22:04:56.839965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.839871 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b"} err="failed to get container status \"a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b\": rpc error: code = NotFound desc = could not find container \"a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b\": container with ID starting with a33ddbe743103c34e140870da3ba0a0c5ce6f5bae79276eb484aa8eafe52094b not found: ID does not exist" Oct 29 22:04:56.859848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.859635 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h"] Oct 29 22:04:56.872656 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.872611 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fd295-8857774fc-dw64h"] Oct 29 22:04:56.925104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.925068 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:04:56.990319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.990226 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ad5968-00d3-4048-a03d-7e5168abd368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ad5968-00d3-4048-a03d-7e5168abd368-plugins-config-volume\") pod \"b6ad5968-00d3-4048-a03d-7e5168abd368\" (UID: \"b6ad5968-00d3-4048-a03d-7e5168abd368\") " Oct 29 22:04:56.990803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.990364 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ad5968-00d3-4048-a03d-7e5168abd368-kube-api-access-q4gjz\" (UniqueName: \"kubernetes.io/projected/b6ad5968-00d3-4048-a03d-7e5168abd368-kube-api-access-q4gjz\") pod \"b6ad5968-00d3-4048-a03d-7e5168abd368\" (UID: \"b6ad5968-00d3-4048-a03d-7e5168abd368\") " Oct 29 22:04:56.992364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.992270 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ad5968-00d3-4048-a03d-7e5168abd368-plugins-config-volume" pod "b6ad5968-00d3-4048-a03d-7e5168abd368" (UID: "b6ad5968-00d3-4048-a03d-7e5168abd368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:04:56.996494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:56.996455 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ad5968-00d3-4048-a03d-7e5168abd368-kube-api-access-q4gjz" pod "b6ad5968-00d3-4048-a03d-7e5168abd368" (UID: "b6ad5968-00d3-4048-a03d-7e5168abd368"). InnerVolumeSpecName "kube-api-access-q4gjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:04:57.108137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.101690 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:57.109897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.109768 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ad5968-00d3-4048-a03d-7e5168abd368-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:57.109897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.109809 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4gjz\" (UniqueName: \"kubernetes.io/projected/b6ad5968-00d3-4048-a03d-7e5168abd368-kube-api-access-q4gjz\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:04:57.281191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.278087 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:57.532528 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.532400 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:57.706741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.706697 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:57.734696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.734658 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" event={"ID":"b6ad5968-00d3-4048-a03d-7e5168abd368","Type":"ContainerDied","Data":"df5c4f0c9d9c20f8f36c985721565ac48ceee2f2839add29306bdf7d47d448cf"} Oct 29 22:04:57.734851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.734719 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv" Oct 29 22:04:57.734851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.734730 4952 scope.go:122] "RemoveContainer" containerID="3a74a8fc58196b948e4c8bbdd75ed94ab8faf8f04b86ce6df3d3634d8cdb9588" Oct 29 22:04:57.793004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.792894 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv"] Oct 29 22:04:57.804392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:57.804351 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fd295-986d46bc9-zmzdv"] Oct 29 22:04:58.037949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:58.033352 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:58.220936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:58.220820 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:58.400927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:58.400529 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:58.646939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:58.629492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:58.655298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:58.655261 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594282a4-47b3-425b-8f9e-1ffac4f2cdc6" path="/var/lib/kubelet/pods/594282a4-47b3-425b-8f9e-1ffac4f2cdc6/volumes" Oct 29 22:04:58.656065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:58.656039 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ad5968-00d3-4048-a03d-7e5168abd368" path="/var/lib/kubelet/pods/b6ad5968-00d3-4048-a03d-7e5168abd368/volumes" Oct 29 22:04:58.802186 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:58.802153 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:04:59.611237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:04:59.601458 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:01.937238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:01.937196 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:02.238926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:02.238812 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:02.414427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:02.414392 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:04.047537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:04.047501 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:04.265945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:04.264521 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:05.989605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:05.989569 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:06.183275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:06.183232 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:06.392179 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:06.373990 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:06.970927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:06.970123 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:07.526975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:07.526126 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:05:07.714889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:07.714853 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:07.916242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:07.916202 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:08.099927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:08.099804 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:08.274446 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:08.272144 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:08.448448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:08.448408 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:10.234974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:10.226478 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:12.078534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:12.078497 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:12.598984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:12.598941 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:12.798891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:12.798854 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:14.157443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:14.157408 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:14.417629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:14.417539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:14.602690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:14.602657 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:15.534643 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:05:15.534603 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:05:15.541787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:15.541756 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:15.717487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:15.717454 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:16.359281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:16.359248 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:16.527227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:16.527191 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:16.695770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:16.695673 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv"] Oct 29 22:05:16.696268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:16.695934 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" podUID="6128724b-db39-4b47-8976-85a9756882a1" containerName="istio-proxy" containerID="cri-o://acbfe18d1e9843ce50445dceac720520ddfc1ec356df6e56a9df1f37ce8ce9de" gracePeriod=30 Oct 29 22:05:16.779669 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:16.779635 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:16.959374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:16.959225 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:17.654801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:17.654763 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:18.190007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:18.189973 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:18.546249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:18.546165 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:19.220112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:19.220074 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:20.046341 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:20.046300 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l"] Oct 29 22:05:20.058544 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:20.058493 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d522daa-ctt2l"] Oct 29 22:05:20.648658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:20.648615 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8a39fb-478f-4a05-b187-a60e82c6d2f2" path="/var/lib/kubelet/pods/da8a39fb-478f-4a05-b187-a60e82c6d2f2/volumes" Oct 29 22:05:21.111624 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:21.111580 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:21.629255 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:21.629217 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:21.805246 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:21.805205 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:22.031592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.031546 4952 generic.go:356] "Generic (PLEG): container finished" podID="6128724b-db39-4b47-8976-85a9756882a1" containerID="acbfe18d1e9843ce50445dceac720520ddfc1ec356df6e56a9df1f37ce8ce9de" exitCode=0 Oct 29 22:05:22.031771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.031737 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" event={"ID":"6128724b-db39-4b47-8976-85a9756882a1","Type":"ContainerDied","Data":"acbfe18d1e9843ce50445dceac720520ddfc1ec356df6e56a9df1f37ce8ce9de"} Oct 29 22:05:22.031834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.031790 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" event={"ID":"6128724b-db39-4b47-8976-85a9756882a1","Type":"ContainerDied","Data":"317b07a270e8795acb0af61be462ae40dbaf0b86b73fd298b99a5fc32e8e8029"} Oct 29 22:05:22.031834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.031815 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317b07a270e8795acb0af61be462ae40dbaf0b86b73fd298b99a5fc32e8e8029" Oct 29 22:05:22.032620 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.032598 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:05:22.175826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.175743 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-data\") pod \"6128724b-db39-4b47-8976-85a9756882a1\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " Oct 29 22:05:22.176034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.175835 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6128724b-db39-4b47-8976-85a9756882a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6128724b-db39-4b47-8976-85a9756882a1-istiod-ca-cert\") pod \"6128724b-db39-4b47-8976-85a9756882a1\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " Oct 29 22:05:22.176034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.175874 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-certs\") pod \"6128724b-db39-4b47-8976-85a9756882a1\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " Oct 29 22:05:22.176034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.175897 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-envoy\") pod \"6128724b-db39-4b47-8976-85a9756882a1\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " Oct 29 22:05:22.176034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.175950 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6128724b-db39-4b47-8976-85a9756882a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6128724b-db39-4b47-8976-85a9756882a1-istio-podinfo\") pod \"6128724b-db39-4b47-8976-85a9756882a1\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " Oct 29 22:05:22.176034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.176023 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-data" pod "6128724b-db39-4b47-8976-85a9756882a1" (UID: "6128724b-db39-4b47-8976-85a9756882a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:22.176321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.176043 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-credential-socket\") pod \"6128724b-db39-4b47-8976-85a9756882a1\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " Oct 29 22:05:22.176321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.176090 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-kube-api-access-h6727\" (UniqueName: \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-kube-api-access-h6727\") pod \"6128724b-db39-4b47-8976-85a9756882a1\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " Oct 29 22:05:22.176321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.176214 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-istio-token\" (UniqueName: \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-istio-token\") pod \"6128724b-db39-4b47-8976-85a9756882a1\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " Oct 29 22:05:22.176321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.176236 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-certs" pod "6128724b-db39-4b47-8976-85a9756882a1" (UID: "6128724b-db39-4b47-8976-85a9756882a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:22.176514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.176343 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-credential-socket" pod "6128724b-db39-4b47-8976-85a9756882a1" (UID: "6128724b-db39-4b47-8976-85a9756882a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:22.176514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.176355 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6128724b-db39-4b47-8976-85a9756882a1-istiod-ca-cert" pod "6128724b-db39-4b47-8976-85a9756882a1" (UID: "6128724b-db39-4b47-8976-85a9756882a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:22.176675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.176653 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-socket\") pod \"6128724b-db39-4b47-8976-85a9756882a1\" (UID: \"6128724b-db39-4b47-8976-85a9756882a1\") " Oct 29 22:05:22.177031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.177001 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-socket" pod "6128724b-db39-4b47-8976-85a9756882a1" (UID: "6128724b-db39-4b47-8976-85a9756882a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:22.177490 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.177465 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.177597 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.177495 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.177597 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.177510 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6128724b-db39-4b47-8976-85a9756882a1-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.177597 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.177523 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.177597 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.177535 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.178337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.178311 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6128724b-db39-4b47-8976-85a9756882a1-istio-podinfo" pod "6128724b-db39-4b47-8976-85a9756882a1" (UID: "6128724b-db39-4b47-8976-85a9756882a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:05:22.178451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.178346 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-istio-token" pod "6128724b-db39-4b47-8976-85a9756882a1" (UID: "6128724b-db39-4b47-8976-85a9756882a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:22.178497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.178480 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-kube-api-access-h6727" pod "6128724b-db39-4b47-8976-85a9756882a1" (UID: "6128724b-db39-4b47-8976-85a9756882a1"). InnerVolumeSpecName "kube-api-access-h6727". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:22.178497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.178486 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-envoy" pod "6128724b-db39-4b47-8976-85a9756882a1" (UID: "6128724b-db39-4b47-8976-85a9756882a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:05:22.278751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.278718 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.278751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.278747 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6128724b-db39-4b47-8976-85a9756882a1-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.278751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.278757 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6128724b-db39-4b47-8976-85a9756882a1-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:22.279024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:22.278766 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6727\" (UniqueName: \"kubernetes.io/projected/6128724b-db39-4b47-8976-85a9756882a1-kube-api-access-h6727\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:23.043377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:23.043302 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv" Oct 29 22:05:23.114142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:23.114113 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv"] Oct 29 22:05:23.126633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:23.126600 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f8867e7d-openshift-default-55b6cbbc77-r2lcv"] Oct 29 22:05:23.363004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:23.362962 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:23.537671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:23.537636 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:23.845285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:23.845246 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:24.645283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:24.645250 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6128724b-db39-4b47-8976-85a9756882a1" path="/var/lib/kubelet/pods/6128724b-db39-4b47-8976-85a9756882a1/volumes" Oct 29 22:05:25.910081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:25.909988 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:27.594967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:27.594030 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:29.304489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:29.304458 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:29.480807 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:29.480764 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:29.660857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:29.660822 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:30.803192 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:30.803154 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt"] Oct 29 22:05:30.803688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:30.803364 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" podUID="522ed822-e84d-4351-83de-1c123dc4de04" containerName="maas-api" containerID="cri-o://1026b364921ddb963832f875c41d194e609b0559fc6542f66792368d6b2c118b" gracePeriod=30 Oct 29 22:05:30.925400 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:30.925365 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:31.122257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.122179 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:31.127885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.127528 4952 generic.go:356] "Generic (PLEG): container finished" podID="522ed822-e84d-4351-83de-1c123dc4de04" containerID="1026b364921ddb963832f875c41d194e609b0559fc6542f66792368d6b2c118b" exitCode=0 Oct 29 22:05:31.127885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.127809 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" event={"ID":"522ed822-e84d-4351-83de-1c123dc4de04","Type":"ContainerDied","Data":"1026b364921ddb963832f875c41d194e609b0559fc6542f66792368d6b2c118b"} Oct 29 22:05:31.186666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.186637 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:05:31.277443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.277203 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522ed822-e84d-4351-83de-1c123dc4de04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522ed822-e84d-4351-83de-1c123dc4de04-openshift-service-ca\") pod \"522ed822-e84d-4351-83de-1c123dc4de04\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " Oct 29 22:05:31.277443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.277343 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-metrics-certs\") pod \"522ed822-e84d-4351-83de-1c123dc4de04\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " Oct 29 22:05:31.277443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.277409 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522ed822-e84d-4351-83de-1c123dc4de04-kube-api-access-gscm4\" (UniqueName: \"kubernetes.io/projected/522ed822-e84d-4351-83de-1c123dc4de04-kube-api-access-gscm4\") pod \"522ed822-e84d-4351-83de-1c123dc4de04\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " Oct 29 22:05:31.277746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.277547 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-maas-api-tls\") pod \"522ed822-e84d-4351-83de-1c123dc4de04\" (UID: \"522ed822-e84d-4351-83de-1c123dc4de04\") " Oct 29 22:05:31.279731 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.279697 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522ed822-e84d-4351-83de-1c123dc4de04-openshift-service-ca" pod "522ed822-e84d-4351-83de-1c123dc4de04" (UID: "522ed822-e84d-4351-83de-1c123dc4de04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:31.281568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.281537 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-metrics-certs" pod "522ed822-e84d-4351-83de-1c123dc4de04" (UID: "522ed822-e84d-4351-83de-1c123dc4de04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:05:31.281672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.281649 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-maas-api-tls" pod "522ed822-e84d-4351-83de-1c123dc4de04" (UID: "522ed822-e84d-4351-83de-1c123dc4de04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:05:31.282347 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.282314 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522ed822-e84d-4351-83de-1c123dc4de04-kube-api-access-gscm4" pod "522ed822-e84d-4351-83de-1c123dc4de04" (UID: "522ed822-e84d-4351-83de-1c123dc4de04"). InnerVolumeSpecName "kube-api-access-gscm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:31.306805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.306776 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:31.380422 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.380221 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:31.380422 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.380259 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522ed822-e84d-4351-83de-1c123dc4de04-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:31.380422 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.380275 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/522ed822-e84d-4351-83de-1c123dc4de04-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:31.380422 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.380290 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gscm4\" (UniqueName: \"kubernetes.io/projected/522ed822-e84d-4351-83de-1c123dc4de04-kube-api-access-gscm4\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:31.519273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.519234 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:31.695695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.693954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:31.862126 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:31.862093 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:32.161259 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:32.161183 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" event={"ID":"522ed822-e84d-4351-83de-1c123dc4de04","Type":"ContainerDied","Data":"6dbc70a6b9797731fab2e4e3416579b8f3a1447322a63ebdc58974dfd62de4f6"} Oct 29 22:05:32.161608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:32.161587 4952 scope.go:122] "RemoveContainer" containerID="1026b364921ddb963832f875c41d194e609b0559fc6542f66792368d6b2c118b" Oct 29 22:05:32.162876 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:32.161796 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt" Oct 29 22:05:32.218922 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:32.218876 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt"] Oct 29 22:05:32.229941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:32.229889 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d9f8c0-7b79d84bdc-8xbbt"] Oct 29 22:05:32.231502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:32.231468 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:32.646797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:32.646761 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522ed822-e84d-4351-83de-1c123dc4de04" path="/var/lib/kubelet/pods/522ed822-e84d-4351-83de-1c123dc4de04/volumes" Oct 29 22:05:33.537155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:33.536791 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:35.051234 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:35.051167 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:35.345710 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:35.345677 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:35.713981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:35.708327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:35.890260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:35.890223 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:36.831704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:36.831665 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5"] Oct 29 22:05:36.832500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:36.832467 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" podUID="9dc98065-570b-41af-be23-5888293bda1f" containerName="maas-api" containerID="cri-o://094faf344b24901b23b8acead9cf312d1742691f4d88ec70e27b8f7066d57a26" gracePeriod=30 Oct 29 22:05:36.955685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:36.955656 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:37.217856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.212880 4952 generic.go:356] "Generic (PLEG): container finished" podID="9dc98065-570b-41af-be23-5888293bda1f" containerID="094faf344b24901b23b8acead9cf312d1742691f4d88ec70e27b8f7066d57a26" exitCode=0 Oct 29 22:05:37.225140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.224879 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" event={"ID":"9dc98065-570b-41af-be23-5888293bda1f","Type":"ContainerDied","Data":"094faf344b24901b23b8acead9cf312d1742691f4d88ec70e27b8f7066d57a26"} Oct 29 22:05:37.412649 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.412233 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:05:37.595978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.595939 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc98065-570b-41af-be23-5888293bda1f-kube-api-access-24h6w\" (UniqueName: \"kubernetes.io/projected/9dc98065-570b-41af-be23-5888293bda1f-kube-api-access-24h6w\") pod \"9dc98065-570b-41af-be23-5888293bda1f\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " Oct 29 22:05:37.596183 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.596061 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-maas-api-tls\") pod \"9dc98065-570b-41af-be23-5888293bda1f\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " Oct 29 22:05:37.596268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.596130 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc98065-570b-41af-be23-5888293bda1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc98065-570b-41af-be23-5888293bda1f-openshift-service-ca\") pod \"9dc98065-570b-41af-be23-5888293bda1f\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " Oct 29 22:05:37.596353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.596311 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-metrics-certs\") pod \"9dc98065-570b-41af-be23-5888293bda1f\" (UID: \"9dc98065-570b-41af-be23-5888293bda1f\") " Oct 29 22:05:37.601939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.598434 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc98065-570b-41af-be23-5888293bda1f-openshift-service-ca" pod "9dc98065-570b-41af-be23-5888293bda1f" (UID: "9dc98065-570b-41af-be23-5888293bda1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:05:37.602759 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.602715 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-maas-api-tls" pod "9dc98065-570b-41af-be23-5888293bda1f" (UID: "9dc98065-570b-41af-be23-5888293bda1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:05:37.602987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.602963 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc98065-570b-41af-be23-5888293bda1f-kube-api-access-24h6w" pod "9dc98065-570b-41af-be23-5888293bda1f" (UID: "9dc98065-570b-41af-be23-5888293bda1f"). InnerVolumeSpecName "kube-api-access-24h6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:05:37.635359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.635310 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-metrics-certs" pod "9dc98065-570b-41af-be23-5888293bda1f" (UID: "9dc98065-570b-41af-be23-5888293bda1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:05:37.700252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.699500 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24h6w\" (UniqueName: \"kubernetes.io/projected/9dc98065-570b-41af-be23-5888293bda1f-kube-api-access-24h6w\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:37.700252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.700128 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:37.700252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.700157 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc98065-570b-41af-be23-5888293bda1f-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:37.700252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.700171 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc98065-570b-41af-be23-5888293bda1f-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:05:37.994116 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:37.994077 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:38.186460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:38.186354 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:38.236568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:38.236525 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" Oct 29 22:05:38.236780 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:38.236701 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5" event={"ID":"9dc98065-570b-41af-be23-5888293bda1f","Type":"ContainerDied","Data":"c104d948ba18e3ffb9431b377b97539bff6fc10302323b8278e86da05669a9e6"} Oct 29 22:05:38.236780 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:38.236756 4952 scope.go:122] "RemoveContainer" containerID="094faf344b24901b23b8acead9cf312d1742691f4d88ec70e27b8f7066d57a26" Oct 29 22:05:38.294966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:38.294043 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5"] Oct 29 22:05:38.309979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:38.308851 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-296d19-856b97786f-n7mz5"] Oct 29 22:05:38.374771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:38.374738 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:38.646952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:38.646886 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc98065-570b-41af-be23-5888293bda1f" path="/var/lib/kubelet/pods/9dc98065-570b-41af-be23-5888293bda1f/volumes" Oct 29 22:05:39.624642 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:39.624608 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:39.801625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:39.801587 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:40.085935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:40.076400 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:40.374984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:40.374472 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:41.542395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:41.542360 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:41.798526 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:41.798437 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:41.981985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:41.981953 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:42.224367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:42.224326 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:42.402608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:42.402577 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:44.259847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:44.259808 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:45.379645 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:45.379605 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:45.528877 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:05:45.528818 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:05:45.878804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:45.878767 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:46.238338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:46.238160 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:46.410968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:46.410894 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:46.611717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:46.611683 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:46.785079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:46.785037 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:47.239483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:47.239443 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:47.426955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:47.426920 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:47.486938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:47.480857 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:05:47.972573 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:47.972535 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:48.194065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:48.194028 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:48.398927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:48.398572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:48.579171 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:48.579135 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:49.970470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:49.970435 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:50.148337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:50.148301 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:51.788505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:51.788457 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:53.374684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:53.374646 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:53.552305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:53.552266 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:54.629664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:54.629631 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:55.125429 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:55.125388 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:55.312296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:55.312238 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:55.740312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:55.740265 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:55.909035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:55.908999 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:56.095169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:56.095132 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:56.274976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:56.274883 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:57.972273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:57.972233 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:58.857846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:58.857813 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:59.350520 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:59.350467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:59.549856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:59.549820 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:05:59.787379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:05:59.787286 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:01.760625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:01.760577 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:03.688463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:03.688429 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:03.915234 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:03.915201 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:04.206847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:04.206812 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:04.424458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:04.424423 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:05.523621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:05.523577 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:05.875161 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:05.875126 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:06.063466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:06.063427 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:07.400899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:07.400861 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:08.073868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:08.073830 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:09.210921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:09.210848 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:09.444454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:09.444412 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:11.166220 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:11.166184 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:13.101390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:13.101349 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:14.297462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:14.297414 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:14.704332 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:14.704297 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:14.890325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:14.890287 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:15.095264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:15.095232 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:15.531782 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:06:15.531685 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:06:17.299997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:17.299954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:17.472975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:17.472895 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:18.779995 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:18.769283 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z"] Oct 29 22:06:18.793831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:18.793792 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" podUID="51468c4a-574a-488c-9b46-0876a15df246" containerName="maas-api" containerID="cri-o://3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc" gracePeriod=30 Oct 29 22:06:18.875019 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:18.874928 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:19.055150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.054841 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:19.206928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.199968 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:06:19.311618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.309358 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51468c4a-574a-488c-9b46-0876a15df246-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51468c4a-574a-488c-9b46-0876a15df246-openshift-service-ca\") pod \"51468c4a-574a-488c-9b46-0876a15df246\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " Oct 29 22:06:19.311618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.309495 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-maas-api-tls\") pod \"51468c4a-574a-488c-9b46-0876a15df246\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " Oct 29 22:06:19.311618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.309629 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51468c4a-574a-488c-9b46-0876a15df246-kube-api-access-6xpjf\" (UniqueName: \"kubernetes.io/projected/51468c4a-574a-488c-9b46-0876a15df246-kube-api-access-6xpjf\") pod \"51468c4a-574a-488c-9b46-0876a15df246\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " Oct 29 22:06:19.329935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.327640 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51468c4a-574a-488c-9b46-0876a15df246-openshift-service-ca" pod "51468c4a-574a-488c-9b46-0876a15df246" (UID: "51468c4a-574a-488c-9b46-0876a15df246"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:06:19.346925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.345510 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51468c4a-574a-488c-9b46-0876a15df246-kube-api-access-6xpjf" pod "51468c4a-574a-488c-9b46-0876a15df246" (UID: "51468c4a-574a-488c-9b46-0876a15df246"). InnerVolumeSpecName "kube-api-access-6xpjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:06:19.417930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.417593 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-metrics-certs\") pod \"51468c4a-574a-488c-9b46-0876a15df246\" (UID: \"51468c4a-574a-488c-9b46-0876a15df246\") " Oct 29 22:06:19.419350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.418722 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xpjf\" (UniqueName: \"kubernetes.io/projected/51468c4a-574a-488c-9b46-0876a15df246-kube-api-access-6xpjf\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:06:19.419350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.418856 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51468c4a-574a-488c-9b46-0876a15df246-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:06:19.441206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.433028 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-maas-api-tls" pod "51468c4a-574a-488c-9b46-0876a15df246" (UID: "51468c4a-574a-488c-9b46-0876a15df246"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:06:19.441555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.441501 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-metrics-certs" pod "51468c4a-574a-488c-9b46-0876a15df246" (UID: "51468c4a-574a-488c-9b46-0876a15df246"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:06:19.520754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.520726 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:06:19.520979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.520967 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51468c4a-574a-488c-9b46-0876a15df246-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:06:19.858109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.858074 4952 generic.go:356] "Generic (PLEG): container finished" podID="51468c4a-574a-488c-9b46-0876a15df246" containerID="3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc" exitCode=0 Oct 29 22:06:19.858684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.858281 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" Oct 29 22:06:19.858839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.858303 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" event={"ID":"51468c4a-574a-488c-9b46-0876a15df246","Type":"ContainerDied","Data":"3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc"} Oct 29 22:06:19.859013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.858992 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z" event={"ID":"51468c4a-574a-488c-9b46-0876a15df246","Type":"ContainerDied","Data":"103ce889af4244a8aa965766537994069ec31b79ca6a2db4e0d34646b3139989"} Oct 29 22:06:19.859222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.859192 4952 scope.go:122] "RemoveContainer" containerID="3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc" Oct 29 22:06:19.875732 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.875711 4952 scope.go:122] "RemoveContainer" containerID="3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc" Oct 29 22:06:19.876064 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:06:19.876044 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc\": container with ID starting with 3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc not found: ID does not exist" containerID="3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc" Oct 29 22:06:19.876150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.876078 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc"} err="failed to get container status \"3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc\": rpc error: code = NotFound desc = could not find container \"3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc\": container with ID starting with 3731d75bb0776213fdfae098a276cb2bcdadaf490d654f46104da5527edec7bc not found: ID does not exist" Oct 29 22:06:19.919477 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.919446 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z"] Oct 29 22:06:19.932222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:19.932193 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4d79fa-5b6d8857bf-mxd5z"] Oct 29 22:06:20.647251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:20.647216 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51468c4a-574a-488c-9b46-0876a15df246" path="/var/lib/kubelet/pods/51468c4a-574a-488c-9b46-0876a15df246/volumes" Oct 29 22:06:20.803669 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:20.803632 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:21.118943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:21.118889 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:21.310002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:21.308032 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:21.748525 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:21.748486 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:21.911748 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:21.911713 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:23.698723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.698669 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:23.852111 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852076 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4"] Oct 29 22:06:23.852835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852802 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b6ad5968-00d3-4048-a03d-7e5168abd368" containerName="payload-pre-processing" Oct 29 22:06:23.852835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852820 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ad5968-00d3-4048-a03d-7e5168abd368" containerName="payload-pre-processing" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852844 4952 cpu_manager.go:414] "containerMap: removing container" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852851 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852865 4952 cpu_manager.go:414] "containerMap: removing container" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852869 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852875 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6128724b-db39-4b47-8976-85a9756882a1" containerName="istio-proxy" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852879 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6128724b-db39-4b47-8976-85a9756882a1" containerName="istio-proxy" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852889 4952 cpu_manager.go:414] "containerMap: removing container" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852893 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852932 4952 cpu_manager.go:414] "containerMap: removing container" podUID="51468c4a-574a-488c-9b46-0876a15df246" containerName="maas-api" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852938 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51468c4a-574a-488c-9b46-0876a15df246" containerName="maas-api" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852952 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9dc98065-570b-41af-be23-5888293bda1f" containerName="maas-api" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852957 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc98065-570b-41af-be23-5888293bda1f" containerName="maas-api" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852974 4952 cpu_manager.go:414] "containerMap: removing container" podUID="522ed822-e84d-4351-83de-1c123dc4de04" containerName="maas-api" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852978 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522ed822-e84d-4351-83de-1c123dc4de04" containerName="maas-api" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852990 4952 cpu_manager.go:414] "containerMap: removing container" podUID="594282a4-47b3-425b-8f9e-1ffac4f2cdc6" containerName="maas-api" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.852998 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594282a4-47b3-425b-8f9e-1ffac4f2cdc6" containerName="maas-api" Oct 29 22:06:23.853108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.853118 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="51468c4a-574a-488c-9b46-0876a15df246" containerName="maas-api" Oct 29 22:06:23.854103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.853129 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc98065-570b-41af-be23-5888293bda1f" containerName="maas-api" Oct 29 22:06:23.854103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.853138 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="522ed822-e84d-4351-83de-1c123dc4de04" containerName="maas-api" Oct 29 22:06:23.854103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.853148 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="594282a4-47b3-425b-8f9e-1ffac4f2cdc6" containerName="maas-api" Oct 29 22:06:23.854103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.853196 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ad5968-00d3-4048-a03d-7e5168abd368" containerName="payload-pre-processing" Oct 29 22:06:23.854103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.853205 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" Oct 29 22:06:23.854103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.853215 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="35194ac9-c066-4437-b1b1-35defc3e122c" containerName="istio-proxy" Oct 29 22:06:23.854103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.853222 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6128724b-db39-4b47-8976-85a9756882a1" containerName="istio-proxy" Oct 29 22:06:23.867803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.867121 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:23.877825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.877790 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4"] Oct 29 22:06:23.883843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.883816 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fd1d4e-serving-cert\"" Oct 29 22:06:23.893318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.893241 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhk6\" (UniqueName: \"kubernetes.io/projected/ee351cf3-bba3-4a34-a124-f570b8b43a9d-kube-api-access-sbhk6\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:23.893455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.893383 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-metrics-certs\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:23.893455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.893417 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:23.893455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.893439 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee351cf3-bba3-4a34-a124-f570b8b43a9d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:23.994412 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.994323 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhk6\" (UniqueName: \"kubernetes.io/projected/ee351cf3-bba3-4a34-a124-f570b8b43a9d-kube-api-access-sbhk6\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:23.994586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.994415 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-metrics-certs\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:23.994586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.994442 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:23.994586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.994463 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee351cf3-bba3-4a34-a124-f570b8b43a9d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:23.994754 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:06:23.994711 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-serving-cert: secret "maas-api-e2e-ipp-a-fd1d4e-serving-cert" not found Oct 29 22:06:23.994808 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:06:23.994768 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-maas-api-tls podName:ee351cf3-bba3-4a34-a124-f570b8b43a9d nodeName:}" failed. No retries permitted until 2026-10-29 22:06:24.494753597 +0000 UTC m=+1449.969341993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-maas-api-tls") pod "maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" (UID: "ee351cf3-bba3-4a34-a124-f570b8b43a9d") : secret "maas-api-e2e-ipp-a-fd1d4e-serving-cert" not found Oct 29 22:06:23.995410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.995378 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee351cf3-bba3-4a34-a124-f570b8b43a9d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:23.997022 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:23.997001 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-metrics-certs\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:24.020851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:24.020831 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhk6\" (UniqueName: \"kubernetes.io/projected/ee351cf3-bba3-4a34-a124-f570b8b43a9d-kube-api-access-sbhk6\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:24.056738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:24.056704 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:24.235816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:24.235777 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:24.409763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:24.409726 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:24.503285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:24.503250 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:24.505672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:24.505649 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:24.654814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:24.654771 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:24.807014 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:24.799208 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:24.912763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:24.912716 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:25.123787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:25.101825 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:25.446899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:25.446866 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4"] Oct 29 22:06:25.448597 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:06:25.448559 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee351cf3_bba3_4a34_a124_f570b8b43a9d.slice/crio-5237d8e781430870d5f72735127a78a8630c3bda2521ec94e579667be0f9c627 WatchSource:0}: Error finding container 5237d8e781430870d5f72735127a78a8630c3bda2521ec94e579667be0f9c627: Status 404 returned error can't find the container with id 5237d8e781430870d5f72735127a78a8630c3bda2521ec94e579667be0f9c627 Oct 29 22:06:25.920197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:25.920161 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" event={"ID":"ee351cf3-bba3-4a34-a124-f570b8b43a9d","Type":"ContainerStarted","Data":"60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b"} Oct 29 22:06:25.920593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:25.920226 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" event={"ID":"ee351cf3-bba3-4a34-a124-f570b8b43a9d","Type":"ContainerStarted","Data":"5237d8e781430870d5f72735127a78a8630c3bda2521ec94e579667be0f9c627"} Oct 29 22:06:25.920593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:25.920255 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:25.968947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:25.967922 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" podStartSLOduration=2.66287254 podStartE2EDuration="2.967882474s" totalImagesPullingTime="305.009934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:25.962325268 +0000 UTC m=+1451.436913687" watchObservedRunningTime="2026-10-29 22:06:25.967882474 +0000 UTC m=+1451.442470893" Oct 29 22:06:25.986986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:25.983201 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:26.048043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:26.047947 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:06:28.263830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:28.263794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:28.591644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:28.591493 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:28.766958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:28.766891 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:29.957601 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:29.957445 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:30.144353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:30.144321 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:30.335489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:30.335455 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:30.507975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:30.507939 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:30.712496 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:30.712399 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:31.934442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:31.934411 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:06:32.228658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:32.228580 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:33.951600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:33.951555 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:35.547436 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.547400 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq"] Oct 29 22:06:35.548366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.548342 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="925f097f-4d45-4843-b5f6-ef6412b3dfa9" containerName="istio-proxy" Oct 29 22:06:35.551773 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.551748 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.573330 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.573299 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq"] Oct 29 22:06:35.578820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.578784 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-796d71-serving-cert\"" Oct 29 22:06:35.595677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.595642 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-metrics-certs\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.595850 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.595700 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e59696-ddbd-4c04-8fc3-9e0796650c2a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.595850 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.595757 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfk8m\" (UniqueName: \"kubernetes.io/projected/79e59696-ddbd-4c04-8fc3-9e0796650c2a-kube-api-access-nfk8m\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.596165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.596138 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.694836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.694802 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:35.697317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.697287 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-metrics-certs\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.697445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.697329 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e59696-ddbd-4c04-8fc3-9e0796650c2a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.697445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.697365 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfk8m\" (UniqueName: \"kubernetes.io/projected/79e59696-ddbd-4c04-8fc3-9e0796650c2a-kube-api-access-nfk8m\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.697565 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.697494 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.698134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.698113 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e59696-ddbd-4c04-8fc3-9e0796650c2a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.699849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.699829 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-metrics-certs\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.699954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.699869 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.721755 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.721720 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfk8m\" (UniqueName: \"kubernetes.io/projected/79e59696-ddbd-4c04-8fc3-9e0796650c2a-kube-api-access-nfk8m\") pod \"maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:35.864751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.864711 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:35.867036 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:35.866571 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:36.178729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:36.178695 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq"] Oct 29 22:06:36.179011 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:06:36.178984 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e59696_ddbd_4c04_8fc3_9e0796650c2a.slice/crio-2217c4344a1e4d0cdce37ba0e7fa5bfd28bc25a01d8a5e010b413a3fbefa99ce WatchSource:0}: Error finding container 2217c4344a1e4d0cdce37ba0e7fa5bfd28bc25a01d8a5e010b413a3fbefa99ce: Status 404 returned error can't find the container with id 2217c4344a1e4d0cdce37ba0e7fa5bfd28bc25a01d8a5e010b413a3fbefa99ce Oct 29 22:06:37.033238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:37.033195 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" event={"ID":"79e59696-ddbd-4c04-8fc3-9e0796650c2a","Type":"ContainerStarted","Data":"421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c"} Oct 29 22:06:37.033643 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:37.033267 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" event={"ID":"79e59696-ddbd-4c04-8fc3-9e0796650c2a","Type":"ContainerStarted","Data":"2217c4344a1e4d0cdce37ba0e7fa5bfd28bc25a01d8a5e010b413a3fbefa99ce"} Oct 29 22:06:37.033643 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:37.033297 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:37.080425 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:37.080372 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" podStartSLOduration=1.751834506 podStartE2EDuration="2.080358789s" totalImagesPullingTime="328.524283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:37.075603232 +0000 UTC m=+1462.550191647" watchObservedRunningTime="2026-10-29 22:06:37.080358789 +0000 UTC m=+1462.554947206" Oct 29 22:06:37.524248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:37.524210 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:39.178215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:39.175788 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:40.955932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:40.952200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:41.696439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:41.696399 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:41.893779 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:41.893745 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:42.844586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:42.844549 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:43.048993 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:43.048960 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:06:44.702874 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:44.701737 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:45.532677 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:06:45.532635 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:06:46.059813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:46.059780 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:46.242836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:46.242795 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:46.591515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:46.591474 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:46.766096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:46.766065 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:46.952618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:46.952506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:47.473885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:47.473846 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:49.387420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:49.387382 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:51.594319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:51.594285 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:53.419446 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:53.419409 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:55.375970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:55.375871 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:57.642117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:57.642030 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:57.982752 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:57.982644 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:06:59.400364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:06:59.400328 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:00.182387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:00.182351 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:00.613220 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:00.613177 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:07:01.337923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:01.334548 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:01.546930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:01.540467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:01.738007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:01.737405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:01.920299 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:01.920261 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:02.688295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:02.688251 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:02.870870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:02.870833 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:03.217297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:03.217252 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:03.520838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:03.520753 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:04.337209 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:04.337170 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:05.665300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:05.665268 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:05.843153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:05.843113 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:06.440319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:06.440270 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:07.512103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:07.512011 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:07.695536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:07.695499 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:08.228123 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:08.228084 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:08.425753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:08.425700 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:09.091263 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:09.091224 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:09.285326 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:09.285281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:09.455140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:09.455054 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:09.988728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:09.988690 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:10.370686 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:10.370647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:10.630522 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:10.630430 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:11.166516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:11.166482 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:13.148389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:13.148270 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:15.026584 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:15.026537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:15.088841 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:15.088813 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:07:15.092951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:15.092926 4952 scope.go:122] "RemoveContainer" containerID="f90f564e9ab4db4580abae25e693ce3979957ea0397c1023ad3fa5b74ddbf973" Oct 29 22:07:15.110178 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:15.110143 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:07:15.529244 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:07:15.529206 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:07:16.712982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:16.712946 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:16.891661 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:16.891617 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:17.200855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:17.200820 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:17.374533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:17.374492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:17.646942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:17.646886 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:18.231744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:18.231709 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:18.614724 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:18.614691 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:19.233991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:19.233956 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:19.396784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:19.396749 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:20.113873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:20.113830 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:20.292671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:20.292610 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:21.358942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:21.358888 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:23.506518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:23.506467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:24.854313 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:24.854252 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:25.032491 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:25.031431 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:25.594896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:25.594846 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:26.737896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:26.737856 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:26.940468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:26.940343 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:28.055896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:28.055865 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:29.706883 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:29.706844 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:31.046663 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.046630 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv"] Oct 29 22:07:31.047449 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.047414 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" podUID="1a9fa251-15d8-4089-9a97-1e290f122b60" containerName="maas-api" containerID="cri-o://0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4" gracePeriod=30 Oct 29 22:07:31.183684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.183601 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:31.402501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.402467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:31.459367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.459340 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:07:31.597161 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.597132 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-metrics-certs\") pod \"1a9fa251-15d8-4089-9a97-1e290f122b60\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " Oct 29 22:07:31.597330 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.597202 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-maas-api-tls\") pod \"1a9fa251-15d8-4089-9a97-1e290f122b60\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " Oct 29 22:07:31.597330 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.597309 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9fa251-15d8-4089-9a97-1e290f122b60-kube-api-access-v8h8r\" (UniqueName: \"kubernetes.io/projected/1a9fa251-15d8-4089-9a97-1e290f122b60-kube-api-access-v8h8r\") pod \"1a9fa251-15d8-4089-9a97-1e290f122b60\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " Oct 29 22:07:31.597502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.597475 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9fa251-15d8-4089-9a97-1e290f122b60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9fa251-15d8-4089-9a97-1e290f122b60-openshift-service-ca\") pod \"1a9fa251-15d8-4089-9a97-1e290f122b60\" (UID: \"1a9fa251-15d8-4089-9a97-1e290f122b60\") " Oct 29 22:07:31.598067 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.598038 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9fa251-15d8-4089-9a97-1e290f122b60-openshift-service-ca" pod "1a9fa251-15d8-4089-9a97-1e290f122b60" (UID: "1a9fa251-15d8-4089-9a97-1e290f122b60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:31.598336 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.598314 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9fa251-15d8-4089-9a97-1e290f122b60-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:31.600063 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.600038 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9fa251-15d8-4089-9a97-1e290f122b60-kube-api-access-v8h8r" pod "1a9fa251-15d8-4089-9a97-1e290f122b60" (UID: "1a9fa251-15d8-4089-9a97-1e290f122b60"). InnerVolumeSpecName "kube-api-access-v8h8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:31.600161 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.600035 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-maas-api-tls" pod "1a9fa251-15d8-4089-9a97-1e290f122b60" (UID: "1a9fa251-15d8-4089-9a97-1e290f122b60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:31.600636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.600561 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-metrics-certs" pod "1a9fa251-15d8-4089-9a97-1e290f122b60" (UID: "1a9fa251-15d8-4089-9a97-1e290f122b60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:07:31.631239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.631207 4952 generic.go:356] "Generic (PLEG): container finished" podID="1a9fa251-15d8-4089-9a97-1e290f122b60" containerID="0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4" exitCode=0 Oct 29 22:07:31.631361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.631269 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" Oct 29 22:07:31.631361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.631293 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" event={"ID":"1a9fa251-15d8-4089-9a97-1e290f122b60","Type":"ContainerDied","Data":"0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4"} Oct 29 22:07:31.631361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.631356 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv" event={"ID":"1a9fa251-15d8-4089-9a97-1e290f122b60","Type":"ContainerDied","Data":"e9bffe3887b52dfd8cee7fc4f60707b0ecd8f2fe66428e70b08b9ee256111e5b"} Oct 29 22:07:31.631550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.631388 4952 scope.go:122] "RemoveContainer" containerID="0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4" Oct 29 22:07:31.645355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.645334 4952 scope.go:122] "RemoveContainer" containerID="0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4" Oct 29 22:07:31.648576 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:07:31.648544 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4\": container with ID starting with 0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4 not found: ID does not exist" containerID="0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4" Oct 29 22:07:31.648695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.648591 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4"} err="failed to get container status \"0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4\": rpc error: code = NotFound desc = could not find container \"0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4\": container with ID starting with 0f612016061d12a486293ba0994e83bc29787f7f50fa68a4ffe8f09f3ab6f9c4 not found: ID does not exist" Oct 29 22:07:31.706987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.702097 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:31.706987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.702155 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9fa251-15d8-4089-9a97-1e290f122b60-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:31.706987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.702174 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8h8r\" (UniqueName: \"kubernetes.io/projected/1a9fa251-15d8-4089-9a97-1e290f122b60-kube-api-access-v8h8r\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:31.762937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.762450 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:31.831967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.831088 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv"] Oct 29 22:07:31.858958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.845218 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-442dc2-79478956cc-r22hv"] Oct 29 22:07:31.953843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:31.946699 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:32.112027 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:32.111989 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:32.646917 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:32.646877 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9fa251-15d8-4089-9a97-1e290f122b60" path="/var/lib/kubelet/pods/1a9fa251-15d8-4089-9a97-1e290f122b60/volumes" Oct 29 22:07:32.814587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:32.814549 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:33.395333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:33.395295 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:33.563070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:33.563035 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:34.643263 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:34.643228 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:36.688403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:36.687229 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:37.649444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:37.649400 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:38.513324 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:38.513282 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:40.388287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:40.388249 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:40.577186 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:40.577151 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:40.763314 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:40.763228 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4"] Oct 29 22:07:40.763810 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:40.763771 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" containerID="cri-o://710abeab757b8e349a2a565f0850be918338fa0b2ebd2c4237fe83c59163144b" gracePeriod=30 Oct 29 22:07:40.901259 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:40.901222 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:41.241835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:41.241798 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:42.401025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.400991 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd"] Oct 29 22:07:42.401664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.401645 4952 cpu_manager.go:414] "containerMap: removing container" podUID="1a9fa251-15d8-4089-9a97-1e290f122b60" containerName="maas-api" Oct 29 22:07:42.401664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.401665 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9fa251-15d8-4089-9a97-1e290f122b60" containerName="maas-api" Oct 29 22:07:42.401840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.401829 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9fa251-15d8-4089-9a97-1e290f122b60" containerName="maas-api" Oct 29 22:07:42.405755 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.405736 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.435216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.435188 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd"] Oct 29 22:07:42.442099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.442070 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c53cc8-serving-cert\"" Oct 29 22:07:42.506632 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.506598 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.506833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.506676 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-metrics-certs\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.506833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.506807 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.507011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.506973 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97ppx\" (UniqueName: \"kubernetes.io/projected/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-kube-api-access-97ppx\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.566264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.566230 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:42.613955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.608926 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97ppx\" (UniqueName: \"kubernetes.io/projected/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-kube-api-access-97ppx\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.613955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.609116 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.613955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.609211 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-metrics-certs\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.613955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.609325 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.613955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.611598 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.615296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.615269 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.635147 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.635116 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-metrics-certs\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.657973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.654329 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97ppx\" (UniqueName: \"kubernetes.io/projected/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-kube-api-access-97ppx\") pod \"maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.724978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.720194 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:42.742741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.742537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:42.916670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:42.916556 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:43.066384 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:07:43.066349 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e08b91f_ac58_4a89_acf2_1c3ad0f886e0.slice/crio-133f77ddb4e936c1a123bacfa17145caf48600b0b536af3cdce48b5ec19ce309 WatchSource:0}: Error finding container 133f77ddb4e936c1a123bacfa17145caf48600b0b536af3cdce48b5ec19ce309: Status 404 returned error can't find the container with id 133f77ddb4e936c1a123bacfa17145caf48600b0b536af3cdce48b5ec19ce309 Oct 29 22:07:43.067174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:43.067142 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd"] Oct 29 22:07:43.876604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:43.876568 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" event={"ID":"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0","Type":"ContainerStarted","Data":"e6f02fa734ea0246aeef51fe6661a28f25ad51c7079fb8b39b9504031e209258"} Oct 29 22:07:43.877136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:43.876631 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" event={"ID":"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0","Type":"ContainerStarted","Data":"133f77ddb4e936c1a123bacfa17145caf48600b0b536af3cdce48b5ec19ce309"} Oct 29 22:07:43.877136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:43.877067 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:43.921379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:43.920523 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" podStartSLOduration=1.539181448 podStartE2EDuration="1.920507209s" totalImagesPullingTime="381.325761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:43.920076263 +0000 UTC m=+1529.394664684" watchObservedRunningTime="2026-10-29 22:07:43.920507209 +0000 UTC m=+1529.395095628" Oct 29 22:07:44.645937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:44.637206 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:45.529025 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:07:45.528989 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:07:45.897658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:45.897627 4952 generic.go:356] "Generic (PLEG): container finished" podID="184e1774-f710-4a0c-a816-af20d9032d86" containerID="710abeab757b8e349a2a565f0850be918338fa0b2ebd2c4237fe83c59163144b" exitCode=0 Oct 29 22:07:45.897852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:45.897711 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" event={"ID":"184e1774-f710-4a0c-a816-af20d9032d86","Type":"ContainerDied","Data":"710abeab757b8e349a2a565f0850be918338fa0b2ebd2c4237fe83c59163144b"} Oct 29 22:07:45.897852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:45.897783 4952 scope.go:122] "RemoveContainer" containerID="4d532fb60a6fa0b8433c1b99a39cd7f4efe3de1df385498cd93ec4eff11ff377" Oct 29 22:07:46.082472 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.082446 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:07:46.192946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.192854 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-envoy\") pod \"184e1774-f710-4a0c-a816-af20d9032d86\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " Oct 29 22:07:46.193140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.192998 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184e1774-f710-4a0c-a816-af20d9032d86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/184e1774-f710-4a0c-a816-af20d9032d86-istiod-ca-cert\") pod \"184e1774-f710-4a0c-a816-af20d9032d86\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " Oct 29 22:07:46.193140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.193037 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-socket\") pod \"184e1774-f710-4a0c-a816-af20d9032d86\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " Oct 29 22:07:46.193277 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.193175 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-credential-socket\") pod \"184e1774-f710-4a0c-a816-af20d9032d86\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " Oct 29 22:07:46.193277 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.193249 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/184e1774-f710-4a0c-a816-af20d9032d86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/184e1774-f710-4a0c-a816-af20d9032d86-istio-podinfo\") pod \"184e1774-f710-4a0c-a816-af20d9032d86\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " Oct 29 22:07:46.193391 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.193344 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-certs\") pod \"184e1774-f710-4a0c-a816-af20d9032d86\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " Oct 29 22:07:46.193507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.193488 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-istio-token\" (UniqueName: \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-istio-token\") pod \"184e1774-f710-4a0c-a816-af20d9032d86\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " Oct 29 22:07:46.193576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.193532 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-data\") pod \"184e1774-f710-4a0c-a816-af20d9032d86\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " Oct 29 22:07:46.193635 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.193604 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-kube-api-access-kqnzl\" (UniqueName: \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-kube-api-access-kqnzl\") pod \"184e1774-f710-4a0c-a816-af20d9032d86\" (UID: \"184e1774-f710-4a0c-a816-af20d9032d86\") " Oct 29 22:07:46.194056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.194018 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-socket" pod "184e1774-f710-4a0c-a816-af20d9032d86" (UID: "184e1774-f710-4a0c-a816-af20d9032d86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:07:46.194177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.194154 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-credential-socket" pod "184e1774-f710-4a0c-a816-af20d9032d86" (UID: "184e1774-f710-4a0c-a816-af20d9032d86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:07:46.194419 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.194395 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-data" pod "184e1774-f710-4a0c-a816-af20d9032d86" (UID: "184e1774-f710-4a0c-a816-af20d9032d86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:07:46.194505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.194482 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184e1774-f710-4a0c-a816-af20d9032d86-istiod-ca-cert" pod "184e1774-f710-4a0c-a816-af20d9032d86" (UID: "184e1774-f710-4a0c-a816-af20d9032d86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:07:46.194712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.194692 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.194783 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.194717 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/184e1774-f710-4a0c-a816-af20d9032d86-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.194783 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.194732 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.194783 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.194746 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.195494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.195467 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-certs" pod "184e1774-f710-4a0c-a816-af20d9032d86" (UID: "184e1774-f710-4a0c-a816-af20d9032d86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:07:46.197182 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.197105 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-envoy" pod "184e1774-f710-4a0c-a816-af20d9032d86" (UID: "184e1774-f710-4a0c-a816-af20d9032d86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:07:46.198746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.198711 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-kube-api-access-kqnzl" pod "184e1774-f710-4a0c-a816-af20d9032d86" (UID: "184e1774-f710-4a0c-a816-af20d9032d86"). InnerVolumeSpecName "kube-api-access-kqnzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:46.198889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.198867 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-istio-token" pod "184e1774-f710-4a0c-a816-af20d9032d86" (UID: "184e1774-f710-4a0c-a816-af20d9032d86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:07:46.199739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.199712 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/184e1774-f710-4a0c-a816-af20d9032d86-istio-podinfo" pod "184e1774-f710-4a0c-a816-af20d9032d86" (UID: "184e1774-f710-4a0c-a816-af20d9032d86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:07:46.296615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.296579 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqnzl\" (UniqueName: \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-kube-api-access-kqnzl\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.296615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.296612 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.296615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.296623 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/184e1774-f710-4a0c-a816-af20d9032d86-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.296847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.296631 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/184e1774-f710-4a0c-a816-af20d9032d86-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.296847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.296639 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/184e1774-f710-4a0c-a816-af20d9032d86-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:07:46.343292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.343254 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:46.906640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.906560 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" Oct 29 22:07:46.906640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.906600 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4" event={"ID":"184e1774-f710-4a0c-a816-af20d9032d86","Type":"ContainerDied","Data":"d48892df537015dff12a40e67537440a61c11c4ba40c5fb6a20df47877f500c4"} Oct 29 22:07:46.907172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.906676 4952 scope.go:122] "RemoveContainer" containerID="710abeab757b8e349a2a565f0850be918338fa0b2ebd2c4237fe83c59163144b" Oct 29 22:07:46.979090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.979055 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4"] Oct 29 22:07:46.988363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:46.988330 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-442dc2-openshift-default-57fbf697-bjfm4"] Oct 29 22:07:48.051348 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:48.051301 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:48.648795 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:48.648758 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184e1774-f710-4a0c-a816-af20d9032d86" path="/var/lib/kubelet/pods/184e1774-f710-4a0c-a816-af20d9032d86/volumes" Oct 29 22:07:49.858617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:49.858583 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:49.891053 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:49.891025 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:07:51.462158 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:51.462105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:53.125669 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:53.125628 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:54.416212 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:54.416166 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:54.586385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:54.586351 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:55.064596 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:55.064557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:56.997853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:56.997808 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:57.178536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:57.178500 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:57.359366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:57.359329 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:57.558079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:57.557848 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:57.742576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:57.742485 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:58.236923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:58.236877 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:59.617691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:59.617655 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:59.793607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:59.793571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:07:59.974583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:07:59.974497 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:01.688315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:01.688276 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:03.858490 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:03.858441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:05.319271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:05.312975 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:08:05.636742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:05.636707 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:05.839825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:05.839782 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:06.016300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:06.016213 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:08.455578 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:08.455537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:08.679989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:08.679951 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:10.435199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:10.435162 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:12.081764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:12.081721 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:13.729652 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:13.729618 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:15.180417 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:15.180378 4952 scope.go:122] "RemoveContainer" containerID="acbfe18d1e9843ce50445dceac720520ddfc1ec356df6e56a9df1f37ce8ce9de" Oct 29 22:08:15.531260 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:08:15.531164 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:08:15.617253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:15.617205 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:17.107295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:17.107256 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:17.474170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:17.474088 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:19.310361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:19.310313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:20.198377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:20.198343 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:20.377766 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:20.377729 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:21.255367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:21.255327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:22.933899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:22.933855 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:24.663359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:24.663311 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:26.444626 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.444549 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:26.625166 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.625127 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4"] Oct 29 22:08:26.626091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.626067 4952 cpu_manager.go:414] "containerMap: removing container" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" Oct 29 22:08:26.626091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.626090 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" Oct 29 22:08:26.626308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.626294 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" Oct 29 22:08:26.626367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.626318 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" Oct 29 22:08:26.632404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.632378 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.679851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.679820 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4"] Oct 29 22:08:26.688608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.688578 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.689133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.688932 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.689133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.688998 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.689133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.689048 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.689133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.689072 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.689424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.689284 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.689424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.689381 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.689424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.689417 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7mg\" (UniqueName: \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-kube-api-access-xl7mg\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.689587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.689509 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.790965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.790879 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.790965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.790954 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7mg\" (UniqueName: \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-kube-api-access-xl7mg\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.790988 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791042 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791195 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791233 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791278 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791306 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791411 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791462 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791665 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791563 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791665 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791642 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791688 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.791876 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.791857 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.793473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.793450 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.793739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.793720 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.815716 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.815682 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.815818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.815770 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7mg\" (UniqueName: \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-kube-api-access-xl7mg\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:26.975846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:26.975812 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:27.109976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:27.105847 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:27.284004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:27.283969 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:27.305068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:27.305040 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4"] Oct 29 22:08:27.305603 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:08:27.305572 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcaf3ea_6b6e_4d3d_865d_55987e5f9446.slice/crio-26e177c868d29b2ebea1b00202ac5cbc57e3867696575165cc26d30071c2e917 WatchSource:0}: Error finding container 26e177c868d29b2ebea1b00202ac5cbc57e3867696575165cc26d30071c2e917: Status 404 returned error can't find the container with id 26e177c868d29b2ebea1b00202ac5cbc57e3867696575165cc26d30071c2e917 Oct 29 22:08:27.308615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:27.308580 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:08:27.308708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:27.308647 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:08:27.308708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:27.308676 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:08:27.446435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:27.446389 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" event={"ID":"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446","Type":"ContainerStarted","Data":"9da487476506349653425611044431bb95a9622a77bffb7c018b53b17c84a814"} Oct 29 22:08:27.446897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:27.446461 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" event={"ID":"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446","Type":"ContainerStarted","Data":"26e177c868d29b2ebea1b00202ac5cbc57e3867696575165cc26d30071c2e917"} Oct 29 22:08:27.494311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:27.494252 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" podStartSLOduration=1.494235325 podStartE2EDuration="1.494235325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:27.493283924 +0000 UTC m=+1572.967872343" watchObservedRunningTime="2026-10-29 22:08:27.494235325 +0000 UTC m=+1572.968823745" Oct 29 22:08:27.976493 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:27.976457 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:28.160978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:28.160293 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:28.977314 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:28.977284 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:08:28.977810 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:28.977334 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:08:29.562958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:29.562921 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:29.760037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:29.749330 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:29.976793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:29.976758 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:08:29.976984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:29.976809 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:08:30.700950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:30.700876 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:30.976862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:30.976783 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:08:30.976862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:30.976836 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:08:31.976705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:31.976380 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:08:31.976705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:31.976441 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:08:32.881499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:32.881461 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:33.005836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:33.005790 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:08:33.006382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:33.005870 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:08:33.986451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:33.981263 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:08:33.986451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:33.981339 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:08:34.980951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:34.979702 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:08:34.980951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:34.979793 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 29 22:08:35.573925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:35.573617 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:36.427080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:36.427046 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:36.428785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:36.427896 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:36.428785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:36.428739 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:08:36.489312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:36.489259 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:36.665065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:36.665019 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:37.861430 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:37.861396 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:38.489618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:38.489575 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:38.659065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:38.659034 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:38.827804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:38.827771 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:39.206934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:39.206822 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:40.968569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:40.968529 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:41.142956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.142636 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:41.320878 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.320794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:41.658636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.658598 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x"] Oct 29 22:08:41.659850 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.659819 4952 cpu_manager.go:414] "containerMap: removing container" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" Oct 29 22:08:41.661102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.661076 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184e1774-f710-4a0c-a816-af20d9032d86" containerName="istio-proxy" Oct 29 22:08:41.666407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.666385 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.699464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.699432 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x"] Oct 29 22:08:41.766448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.766409 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.766642 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.766541 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.766759 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.766737 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.766829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.766793 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.766986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.766962 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxh5d\" (UniqueName: \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-kube-api-access-zxh5d\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.767035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.767009 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.767129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.767115 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.767250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.767229 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.767292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.767264 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.869031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.868990 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.869209 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.869081 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.869209 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.869152 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.869370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.869326 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.869508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.869485 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-certs\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.869593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.869528 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-credential-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.869730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.869707 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.869845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.869815 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-socket\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.870001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.869893 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxh5d\" (UniqueName: \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-kube-api-access-zxh5d\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.870001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.869981 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.870119 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.870053 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.870175 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.870161 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istiod-ca-cert\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.870252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.870175 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.870655 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.870633 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-data\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.872507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.872482 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-podinfo\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.872587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.872553 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-envoy\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.894154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.894125 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-token\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.894275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.894205 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxh5d\" (UniqueName: \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-kube-api-access-zxh5d\") pod \"e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:41.989242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:41.989154 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:42.338941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:42.336266 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x"] Oct 29 22:08:42.345562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:42.345525 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:08:42.345692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:42.345613 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:08:42.345692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:42.345658 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:08:42.604278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:42.604192 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:42.958465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:42.958373 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" event={"ID":"99e98d47-7a58-41d8-8ab2-09b52addd0ce","Type":"ContainerStarted","Data":"e4ef3ad7d6d5e2e036921bba20b359c1735f8dac89406bcff42fd8f9cd5676c3"} Oct 29 22:08:42.958730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:42.958706 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" event={"ID":"99e98d47-7a58-41d8-8ab2-09b52addd0ce","Type":"ContainerStarted","Data":"8b2bf5433102494817cf18958fd0f36680f910d27222721bddd76bbf4375bc8f"} Oct 29 22:08:43.006934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:42.997006 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:43.093930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:43.055450 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" podStartSLOduration=2.05542813 podStartE2EDuration="2.05542813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:43.038521981 +0000 UTC m=+1588.513110412" watchObservedRunningTime="2026-10-29 22:08:43.05542813 +0000 UTC m=+1588.530016549" Oct 29 22:08:43.383950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:43.383849 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:43.554520 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:43.554481 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:43.745983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:43.744488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:43.991498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:43.991461 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:08:43.991691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:43.991514 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:08:44.062552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:44.062451 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:44.990447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:44.990413 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:08:44.990631 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:44.990460 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:08:45.568500 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:08:45.568439 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:08:45.683843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:45.683790 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:45.989682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:45.989651 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:08:45.989851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:45.989702 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:08:46.581637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:46.581593 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:46.762340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:46.757360 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:46.959014 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:46.957866 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:47.041411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:47.041359 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:08:47.041701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:47.041671 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:08:47.684897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:47.684812 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:47.989866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:47.989783 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:08:47.989866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:47.989852 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:08:48.989667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:48.989634 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:08:48.990109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:48.989696 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:08:49.691078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:49.690647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:49.990546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:49.990445 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:08:49.990546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:49.990503 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:08:50.946601 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:50.946561 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:51.027056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:51.027023 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:51.067532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:51.067498 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:51.079663 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:51.079616 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:08:51.124137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:51.120521 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:52.190598 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:52.190557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:52.395185 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:52.395152 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:52.666538 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:52.666496 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:54.255513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:54.255476 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:56.015815 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:56.015772 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:57.720488 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:57.720449 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:58.203645 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:58.203597 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:58.378802 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:58.378765 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:08:59.524158 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:08:59.524118 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:01.239366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:01.239319 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:02.971845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:02.969474 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:03.812022 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:03.811971 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:03.978349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:03.978305 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:05.013842 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.013806 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq"] Oct 29 22:09:05.042992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.042959 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.057523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.057455 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq"] Oct 29 22:09:05.107601 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.107551 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.107796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.107662 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.107796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.107709 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.107935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.107822 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wf5k\" (UniqueName: \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-kube-api-access-4wf5k\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.107935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.107888 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.108050 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.107964 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.108090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.108049 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.108149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.108120 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2658a91b-fbc8-4894-8ce9-3ba063792034-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.108296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.108268 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.209604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.209570 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.209797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.209628 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.209797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.209677 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.209797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.209708 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wf5k\" (UniqueName: \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-kube-api-access-4wf5k\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.209984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.209844 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.209984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.209930 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.210104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.209994 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.210165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.210104 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2658a91b-fbc8-4894-8ce9-3ba063792034-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.210221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.210181 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.210316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.210287 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.210408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.210386 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-data\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.210500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.210407 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-certs\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.210568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.210543 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-credential-socket\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.210826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.210802 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2658a91b-fbc8-4894-8ce9-3ba063792034-istiod-ca-cert\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.212256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.212232 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-envoy\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.212460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.212440 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-podinfo\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.234987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.234950 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-token\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.245459 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.245427 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wf5k\" (UniqueName: \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-kube-api-access-4wf5k\") pod \"e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.358648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.358600 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:05.372625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.372590 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:05.736432 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:09:05.736397 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2658a91b_fbc8_4894_8ce9_3ba063792034.slice/crio-cab1656a305a69004ddd7faa4377073f1354d03f1a8d2cc48803f7c95cebfa9e WatchSource:0}: Error finding container cab1656a305a69004ddd7faa4377073f1354d03f1a8d2cc48803f7c95cebfa9e: Status 404 returned error can't find the container with id cab1656a305a69004ddd7faa4377073f1354d03f1a8d2cc48803f7c95cebfa9e Oct 29 22:09:05.737383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.737359 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq"] Oct 29 22:09:05.739514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.739478 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:09:05.739631 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.739546 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:09:05.739631 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:05.739579 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:09:06.410640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:06.410586 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" event={"ID":"2658a91b-fbc8-4894-8ce9-3ba063792034","Type":"ContainerStarted","Data":"69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1"} Oct 29 22:09:06.411116 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:06.410659 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" event={"ID":"2658a91b-fbc8-4894-8ce9-3ba063792034","Type":"ContainerStarted","Data":"cab1656a305a69004ddd7faa4377073f1354d03f1a8d2cc48803f7c95cebfa9e"} Oct 29 22:09:06.473530 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:06.473475 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" podStartSLOduration=2.473460237 podStartE2EDuration="2.473460237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:06.468026465 +0000 UTC m=+1611.942614884" watchObservedRunningTime="2026-10-29 22:09:06.473460237 +0000 UTC m=+1611.948048654" Oct 29 22:09:07.154280 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:07.154246 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:07.373367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:07.373319 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:07.375187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:07.375159 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:09:07.375316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:07.375207 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:09:08.373574 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:08.373384 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:09:08.373574 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:08.373454 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:09:08.548042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:08.548010 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:09.065932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:09.062573 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:09.374230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:09.374189 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:09:09.375235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:09.375202 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:09:10.373183 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:10.373143 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:09:10.373366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:10.373202 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:09:11.084138 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:11.084109 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:09:11.377361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:11.377277 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:09:11.377535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:11.377367 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:09:11.733420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:11.733329 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:12.373684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:12.373646 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:09:12.374143 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:12.373712 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:09:13.685854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:13.685816 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:13.686413 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:13.686348 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:13.687678 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:13.687645 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:09:14.142486 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:14.142448 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:15.528959 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:09:15.528918 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:09:15.930024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:15.929982 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:17.547404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:17.547264 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:18.389927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:18.389546 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:18.641715 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:18.641620 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:18.834536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:18.834497 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:20.778497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:20.778347 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:22.556310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:22.556270 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:24.208841 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:24.208801 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:25.380676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:25.380631 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:25.599250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:25.599210 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:25.828888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:25.828838 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:26.340975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:26.340758 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:28.164113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:28.164068 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:29.866043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:29.866008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:31.673113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:31.671460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:33.870885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:33.870850 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:35.722675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:35.722633 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:37.552010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:37.551971 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:39.307394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.307357 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4"] Oct 29 22:09:39.308060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.307612 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" podUID="ee351cf3-bba3-4a34-a124-f570b8b43a9d" containerName="maas-api" containerID="cri-o://60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b" gracePeriod=30 Oct 29 22:09:39.431933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.428647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:39.666144 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.665419 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:39.745999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.745969 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:09:39.806930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.804759 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee351cf3-bba3-4a34-a124-f570b8b43a9d-kube-api-access-sbhk6\" (UniqueName: \"kubernetes.io/projected/ee351cf3-bba3-4a34-a124-f570b8b43a9d-kube-api-access-sbhk6\") pod \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " Oct 29 22:09:39.806930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.804837 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-metrics-certs\") pod \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " Oct 29 22:09:39.827135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.823171 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-maas-api-tls\") pod \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " Oct 29 22:09:39.827135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.823359 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee351cf3-bba3-4a34-a124-f570b8b43a9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee351cf3-bba3-4a34-a124-f570b8b43a9d-openshift-service-ca\") pod \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\" (UID: \"ee351cf3-bba3-4a34-a124-f570b8b43a9d\") " Oct 29 22:09:39.827135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.824773 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee351cf3-bba3-4a34-a124-f570b8b43a9d-openshift-service-ca" pod "ee351cf3-bba3-4a34-a124-f570b8b43a9d" (UID: "ee351cf3-bba3-4a34-a124-f570b8b43a9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:09:39.872140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.869778 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:39.872140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.870867 4952 generic.go:356] "Generic (PLEG): container finished" podID="ee351cf3-bba3-4a34-a124-f570b8b43a9d" containerID="60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b" exitCode=0 Oct 29 22:09:39.872140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.871109 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" event={"ID":"ee351cf3-bba3-4a34-a124-f570b8b43a9d","Type":"ContainerDied","Data":"60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b"} Oct 29 22:09:39.872140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.871162 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" event={"ID":"ee351cf3-bba3-4a34-a124-f570b8b43a9d","Type":"ContainerDied","Data":"5237d8e781430870d5f72735127a78a8630c3bda2521ec94e579667be0f9c627"} Oct 29 22:09:39.872140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.871194 4952 scope.go:122] "RemoveContainer" containerID="60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b" Oct 29 22:09:39.872140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.871361 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4" Oct 29 22:09:39.978092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.947966 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee351cf3-bba3-4a34-a124-f570b8b43a9d-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:09:39.978847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.978796 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-metrics-certs" pod "ee351cf3-bba3-4a34-a124-f570b8b43a9d" (UID: "ee351cf3-bba3-4a34-a124-f570b8b43a9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:39.996132 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:39.996072 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-maas-api-tls" pod "ee351cf3-bba3-4a34-a124-f570b8b43a9d" (UID: "ee351cf3-bba3-4a34-a124-f570b8b43a9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:09:40.029750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.025420 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee351cf3-bba3-4a34-a124-f570b8b43a9d-kube-api-access-sbhk6" pod "ee351cf3-bba3-4a34-a124-f570b8b43a9d" (UID: "ee351cf3-bba3-4a34-a124-f570b8b43a9d"). InnerVolumeSpecName "kube-api-access-sbhk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:09:40.063939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.055196 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:09:40.063939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.055235 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbhk6\" (UniqueName: \"kubernetes.io/projected/ee351cf3-bba3-4a34-a124-f570b8b43a9d-kube-api-access-sbhk6\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:09:40.063939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.055250 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee351cf3-bba3-4a34-a124-f570b8b43a9d-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:09:40.115754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.109346 4952 scope.go:122] "RemoveContainer" containerID="60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b" Oct 29 22:09:40.115754 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:09:40.110831 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b\": container with ID starting with 60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b not found: ID does not exist" containerID="60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b" Oct 29 22:09:40.115754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.110873 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b"} err="failed to get container status \"60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b\": rpc error: code = NotFound desc = could not find container \"60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b\": container with ID starting with 60f0c381cdcc1765b410adf49f9aac41e4b1cc75668dc770a442eec4e0e98a0b not found: ID does not exist" Oct 29 22:09:40.198723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.198630 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:40.221815 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.221782 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4"] Oct 29 22:09:40.232679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.232638 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fd1d4e-57f6b586f4-b94h4"] Oct 29 22:09:40.370371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.370254 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:40.547357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.547267 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:40.646574 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.646533 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee351cf3-bba3-4a34-a124-f570b8b43a9d" path="/var/lib/kubelet/pods/ee351cf3-bba3-4a34-a124-f570b8b43a9d/volumes" Oct 29 22:09:40.861534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:40.861477 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:41.962792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:41.962756 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:42.592972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:42.590221 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:42.777753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:42.777603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:43.248380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:43.248345 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:44.739931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:44.739790 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:45.533659 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:09:45.533619 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:09:47.261013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:47.260952 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:47.748393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:47.748342 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:47.921643 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:47.921607 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:48.188261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:48.188226 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:49.585663 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:49.585612 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:51.026565 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:51.026524 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:51.197371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:51.197338 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:51.442502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:51.442465 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:51.643792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:51.643753 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:52.987370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:52.987331 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:53.203860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:53.203818 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:53.535943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:53.533198 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:53.860371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:53.860327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:55.202180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:55.202139 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:56.816745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:56.816655 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:56.995282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:56.995238 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:57.332938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:57.330496 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:58.536769 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:58.536732 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:58.730551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:58.730517 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:09:59.575432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:09:59.575398 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:01.840273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:01.840233 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:03.615020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:03.614984 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:05.332842 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:05.332805 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:07.530118 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:07.530029 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:08.660526 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:08.660491 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:09.261493 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:09.261457 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:09.623455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:09.623412 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:10:10.775934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:10.775882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:10.812443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:10.812408 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd"] Oct 29 22:10:10.813232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:10.813176 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" podUID="4e08b91f-ac58-4a89-acf2-1c3ad0f886e0" containerName="maas-api" containerID="cri-o://e6f02fa734ea0246aeef51fe6661a28f25ad51c7079fb8b39b9504031e209258" gracePeriod=30 Oct 29 22:10:11.139030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:11.123869 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:11.503982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:11.501298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:11.587932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:11.583019 4952 generic.go:356] "Generic (PLEG): container finished" podID="4e08b91f-ac58-4a89-acf2-1c3ad0f886e0" containerID="e6f02fa734ea0246aeef51fe6661a28f25ad51c7079fb8b39b9504031e209258" exitCode=0 Oct 29 22:10:11.587932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:11.583233 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" event={"ID":"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0","Type":"ContainerDied","Data":"e6f02fa734ea0246aeef51fe6661a28f25ad51c7079fb8b39b9504031e209258"} Oct 29 22:10:11.880181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:11.860926 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:12.168007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.165362 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:12.168007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.166045 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:12.623932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.618473 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:12.623932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.620832 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:10:12.810399 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.801355 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" Oct 29 22:10:12.836060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.819657 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-openshift-service-ca\") pod \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " Oct 29 22:10:12.836060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.819837 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-kube-api-access-97ppx\" (UniqueName: \"kubernetes.io/projected/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-kube-api-access-97ppx\") pod \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " Oct 29 22:10:12.836060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.819947 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-maas-api-tls\") pod \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " Oct 29 22:10:12.836060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.819989 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-metrics-certs\") pod \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\" (UID: \"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0\") " Oct 29 22:10:12.867401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.867366 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-openshift-service-ca" pod "4e08b91f-ac58-4a89-acf2-1c3ad0f886e0" (UID: "4e08b91f-ac58-4a89-acf2-1c3ad0f886e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:10:12.869753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.869653 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:12.931967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.931695 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:10:12.949286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.942949 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd" event={"ID":"4e08b91f-ac58-4a89-acf2-1c3ad0f886e0","Type":"ContainerDied","Data":"133f77ddb4e936c1a123bacfa17145caf48600b0b536af3cdce48b5ec19ce309"} Oct 29 22:10:12.949286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.943048 4952 scope.go:122] "RemoveContainer" containerID="e6f02fa734ea0246aeef51fe6661a28f25ad51c7079fb8b39b9504031e209258" Oct 29 22:10:12.973271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.952277 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-kube-api-access-97ppx" pod "4e08b91f-ac58-4a89-acf2-1c3ad0f886e0" (UID: "4e08b91f-ac58-4a89-acf2-1c3ad0f886e0"). InnerVolumeSpecName "kube-api-access-97ppx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:10:12.973271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.960026 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:10:12.978947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.977351 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:10:12.978947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.977722 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:10:12.981129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:12.981081 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-metrics-certs" pod "4e08b91f-ac58-4a89-acf2-1c3ad0f886e0" (UID: "4e08b91f-ac58-4a89-acf2-1c3ad0f886e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:10:13.062999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.024856 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:10:13.062999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.053996 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.062999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.057079 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97ppx\" (UniqueName: \"kubernetes.io/projected/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-kube-api-access-97ppx\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:10:13.062999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.057109 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:10:13.089076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.089035 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-maas-api-tls" pod "4e08b91f-ac58-4a89-acf2-1c3ad0f886e0" (UID: "4e08b91f-ac58-4a89-acf2-1c3ad0f886e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:10:13.163320 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.163283 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:10:13.179126 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.179081 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.194775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.185943 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 22:10:13.276473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.267332 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.276473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.269725 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 29 22:10:13.340262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.340235 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd"] Oct 29 22:10:13.373214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.373173 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 29 22:10:13.373817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.373784 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 22:10:13.392243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.392205 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.421051 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.420999 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.438061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.438001 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 22:10:13.573091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.543205 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.573406 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.543247 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 22:10:13.580016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.579979 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 22:10:13.847222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.847185 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c53cc8-597775b9b8-bffkd"] Oct 29 22:10:13.962601 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.962143 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.965352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.963498 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 22:10:13.997926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.996643 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:13.998932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:13.998131 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:10:14.013973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.003773 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.013973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.004090 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:10:14.069010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.050645 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:10:14.069010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.066597 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 22:10:14.209420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.209330 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 29 22:10:14.270250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.270115 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.279508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.279392 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 22:10:14.378875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.378833 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.379451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.379420 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 29 22:10:14.379893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.379874 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 22:10:14.450899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.450871 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:10:14.452686 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.452664 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:10:14.474053 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.426274 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:14.475679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.475188 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 22:10:14.475679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.475243 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 22:10:14.493083 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.478058 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 22:10:14.540748 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.540706 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 22:10:14.575058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.575029 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 22:10:14.580996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.580961 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:10:14.582046 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.582021 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 22:10:14.601627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.601599 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:10:14.602189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.602167 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 22:10:14.666972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:14.653928 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e08b91f-ac58-4a89-acf2-1c3ad0f886e0" path="/var/lib/kubelet/pods/4e08b91f-ac58-4a89-acf2-1c3ad0f886e0/volumes" Oct 29 22:10:15.395230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:15.394604 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:15.535956 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:10:15.534179 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:10:17.156925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:17.155861 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:19.024704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:19.023556 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:20.516916 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:20.516866 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:20.692221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:20.692185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:20.943180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:20.943145 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:21.110709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:21.110673 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:22.530896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:22.530859 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:22.713095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:22.713040 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:22.968694 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:22.968656 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:23.142514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:23.142478 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:24.649706 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:24.648692 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:26.553399 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:26.553363 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:27.544377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:27.544341 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:27.727351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:27.727306 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:28.933028 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:28.931131 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:29.322972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:29.322869 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:29.510785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:29.510749 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:31.053603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:31.053568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:32.953931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:32.952031 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:34.651424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:34.651386 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:36.391640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:36.391560 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:38.078823 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:38.078790 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:39.722668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:39.722619 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:41.396621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:41.396571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:43.189887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:43.189848 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:44.917547 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:44.917506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:45.529407 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:10:45.529360 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:10:46.700951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:46.700888 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:47.834512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:47.834469 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:49.095280 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:49.095243 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:49.269652 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:49.269603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:49.646623 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:49.646590 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:50.488616 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:50.488579 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:50.586034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:50.585990 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.185934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.147571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:51.204581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.204546 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 29 22:10:51.448039 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.437660 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 22:10:51.507927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.493620 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.507927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.500942 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.507927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.502274 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.507927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.502961 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 29 22:10:51.567942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.555044 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 22:10:51.620174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.620117 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:51.637282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.637237 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:10:51.650037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.649983 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:10:51.728968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:51.726524 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:10:52.139973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.138084 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:10:52.139973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.138149 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.176644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.176582 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 29 22:10:52.221004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.219017 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.471585 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.471476 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.474872 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.474844 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 22:10:52.506931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.506868 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.508794 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.508768 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:10:52.591930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.553595 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.592582 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.553670 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.619584 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.619543 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 22:10:52.646372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.621439 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.646940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.624231 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:10:52.682681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.679941 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 22:10:52.740876 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.740760 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:10:52.794337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.794304 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:52.918385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.880091 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 22:10:52.918385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.882132 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 22:10:52.918385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.882184 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 22:10:52.918385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:52.886123 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:10:53.218929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:53.218580 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:10:55.278395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:55.278353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:56.257008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:56.256963 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:56.447141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:56.447093 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:57.134928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:57.131867 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:57.975027 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:57.974987 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:58.140574 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:58.140528 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:10:58.908092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:10:58.908057 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:00.858036 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:00.857993 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:02.764349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:02.764304 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:04.163551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.163520 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g"] Oct 29 22:11:04.164109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.163737 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" podUID="8a7008d6-5941-4e8b-b0d3-a576a186c8d0" containerName="maas-api" containerID="cri-o://5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc" gracePeriod=30 Oct 29 22:11:04.279027 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.278995 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:04.526993 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.526044 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:04.577560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.577528 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:11:04.664112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.664075 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-kube-api-access-dvmw8\" (UniqueName: \"kubernetes.io/projected/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-kube-api-access-dvmw8\") pod \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " Oct 29 22:11:04.664337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.664193 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-openshift-service-ca\") pod \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " Oct 29 22:11:04.664337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.664229 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-metrics-certs\") pod \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " Oct 29 22:11:04.664337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.664257 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-maas-api-tls\") pod \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\" (UID: \"8a7008d6-5941-4e8b-b0d3-a576a186c8d0\") " Oct 29 22:11:04.669461 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.669411 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-openshift-service-ca" pod "8a7008d6-5941-4e8b-b0d3-a576a186c8d0" (UID: "8a7008d6-5941-4e8b-b0d3-a576a186c8d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:04.677567 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.677528 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-metrics-certs" pod "8a7008d6-5941-4e8b-b0d3-a576a186c8d0" (UID: "8a7008d6-5941-4e8b-b0d3-a576a186c8d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:11:04.693133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.693099 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-maas-api-tls" pod "8a7008d6-5941-4e8b-b0d3-a576a186c8d0" (UID: "8a7008d6-5941-4e8b-b0d3-a576a186c8d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:11:04.700074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.700044 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-kube-api-access-dvmw8" pod "8a7008d6-5941-4e8b-b0d3-a576a186c8d0" (UID: "8a7008d6-5941-4e8b-b0d3-a576a186c8d0"). InnerVolumeSpecName "kube-api-access-dvmw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:04.728487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.728460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:04.935930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.882473 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvmw8\" (UniqueName: \"kubernetes.io/projected/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-kube-api-access-dvmw8\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:04.935930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.882506 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:04.935930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.882521 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:04.935930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:04.882534 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a7008d6-5941-4e8b-b0d3-a576a186c8d0-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:05.122870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.122832 4952 generic.go:356] "Generic (PLEG): container finished" podID="8a7008d6-5941-4e8b-b0d3-a576a186c8d0" containerID="5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc" exitCode=0 Oct 29 22:11:05.123054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.122876 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" event={"ID":"8a7008d6-5941-4e8b-b0d3-a576a186c8d0","Type":"ContainerDied","Data":"5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc"} Oct 29 22:11:05.123054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.122938 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" event={"ID":"8a7008d6-5941-4e8b-b0d3-a576a186c8d0","Type":"ContainerDied","Data":"003bb297fc2d089645d38905a0e06d32cb585a834e9c41fccf253b7fa1633ba0"} Oct 29 22:11:05.123054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.122940 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g" Oct 29 22:11:05.123054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.123007 4952 scope.go:122] "RemoveContainer" containerID="5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc" Oct 29 22:11:05.135018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.134997 4952 scope.go:122] "RemoveContainer" containerID="5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc" Oct 29 22:11:05.135278 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:11:05.135258 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc\": container with ID starting with 5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc not found: ID does not exist" containerID="5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc" Oct 29 22:11:05.135328 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.135287 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc"} err="failed to get container status \"5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc\": rpc error: code = NotFound desc = could not find container \"5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc\": container with ID starting with 5ba9945eaf0d2cf0bb7c680f6752981c2df6e31fab8e2aa4109d23b9a9f7b8dc not found: ID does not exist" Oct 29 22:11:05.160670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.160639 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:05.180404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.180380 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g"] Oct 29 22:11:05.193397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.193343 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fe45b-c4fd9966f-s657g"] Oct 29 22:11:05.331844 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.331808 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:05.499591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.499503 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:05.815998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:05.815845 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:06.646484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:06.646445 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7008d6-5941-4e8b-b0d3-a576a186c8d0" path="/var/lib/kubelet/pods/8a7008d6-5941-4e8b-b0d3-a576a186c8d0/volumes" Oct 29 22:11:07.178952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:07.178917 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9"] Oct 29 22:11:07.179135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:07.179101 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" containerID="cri-o://895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368" gracePeriod=30 Oct 29 22:11:07.190990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:07.190960 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d"] Oct 29 22:11:07.191218 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:07.191192 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" containerID="cri-o://f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3" gracePeriod=30 Oct 29 22:11:07.242768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:07.242734 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:08.764401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:08.764368 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:09.028981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:09.027965 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:10.803286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:10.803250 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:10.972259 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:10.972217 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:11.139407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:11.139350 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl"] Oct 29 22:11:11.139843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:11.139821 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg"] Oct 29 22:11:11.140238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:11.140208 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" containerID="cri-o://ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575" gracePeriod=30 Oct 29 22:11:11.140600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:11.140575 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerName="istio-proxy" containerID="cri-o://6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934" gracePeriod=30 Oct 29 22:11:11.152789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:11.152754 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:12.510494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.510405 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 22:11:12.552610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.552589 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 22:11:12.560764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.560738 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d970934-54ed-4252-b585-1a0d34aeeaa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d970934-54ed-4252-b585-1a0d34aeeaa9-istiod-ca-cert\") pod \"8d970934-54ed-4252-b585-1a0d34aeeaa9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " Oct 29 22:11:12.560875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.560774 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-podinfo\") pod \"8d970934-54ed-4252-b585-1a0d34aeeaa9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " Oct 29 22:11:12.560875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.560808 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-credential-socket\") pod \"9a020b72-5cdf-4826-800c-c43eb3708ec2\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " Oct 29 22:11:12.560875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.560847 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-kube-api-access-rnqrw\" (UniqueName: \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-kube-api-access-rnqrw\") pod \"9a020b72-5cdf-4826-800c-c43eb3708ec2\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " Oct 29 22:11:12.561089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.560896 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a020b72-5cdf-4826-800c-c43eb3708ec2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a020b72-5cdf-4826-800c-c43eb3708ec2-istiod-ca-cert\") pod \"9a020b72-5cdf-4826-800c-c43eb3708ec2\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " Oct 29 22:11:12.561089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561000 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-podinfo\") pod \"9a020b72-5cdf-4826-800c-c43eb3708ec2\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " Oct 29 22:11:12.561089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561033 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-envoy\") pod \"8d970934-54ed-4252-b585-1a0d34aeeaa9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " Oct 29 22:11:12.561249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561145 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-credential-socket\") pod \"8d970934-54ed-4252-b585-1a0d34aeeaa9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " Oct 29 22:11:12.561249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561189 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d970934-54ed-4252-b585-1a0d34aeeaa9-istiod-ca-cert" pod "8d970934-54ed-4252-b585-1a0d34aeeaa9" (UID: "8d970934-54ed-4252-b585-1a0d34aeeaa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:12.561249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561239 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-data\") pod \"8d970934-54ed-4252-b585-1a0d34aeeaa9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " Oct 29 22:11:12.561416 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561270 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-token\" (UniqueName: \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-token\") pod \"8d970934-54ed-4252-b585-1a0d34aeeaa9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " Oct 29 22:11:12.561416 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561323 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-token\" (UniqueName: \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-token\") pod \"9a020b72-5cdf-4826-800c-c43eb3708ec2\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " Oct 29 22:11:12.561416 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561402 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-data\") pod \"9a020b72-5cdf-4826-800c-c43eb3708ec2\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " Oct 29 22:11:12.561587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561510 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-socket\") pod \"9a020b72-5cdf-4826-800c-c43eb3708ec2\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " Oct 29 22:11:12.561587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561543 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-kube-api-access-k5f7m\" (UniqueName: \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-kube-api-access-k5f7m\") pod \"8d970934-54ed-4252-b585-1a0d34aeeaa9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " Oct 29 22:11:12.561698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561628 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-socket\") pod \"8d970934-54ed-4252-b585-1a0d34aeeaa9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " Oct 29 22:11:12.561698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561674 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a020b72-5cdf-4826-800c-c43eb3708ec2-istiod-ca-cert" pod "9a020b72-5cdf-4826-800c-c43eb3708ec2" (UID: "9a020b72-5cdf-4826-800c-c43eb3708ec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:12.561698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561693 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-certs\") pod \"8d970934-54ed-4252-b585-1a0d34aeeaa9\" (UID: \"8d970934-54ed-4252-b585-1a0d34aeeaa9\") " Oct 29 22:11:12.561866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561747 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-credential-socket" pod "8d970934-54ed-4252-b585-1a0d34aeeaa9" (UID: "8d970934-54ed-4252-b585-1a0d34aeeaa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:12.561866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561769 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-certs\") pod \"9a020b72-5cdf-4826-800c-c43eb3708ec2\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " Oct 29 22:11:12.561866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561812 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-envoy\") pod \"9a020b72-5cdf-4826-800c-c43eb3708ec2\" (UID: \"9a020b72-5cdf-4826-800c-c43eb3708ec2\") " Oct 29 22:11:12.562055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.561972 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-data" pod "9a020b72-5cdf-4826-800c-c43eb3708ec2" (UID: "9a020b72-5cdf-4826-800c-c43eb3708ec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:12.562184 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.562153 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-data" pod "8d970934-54ed-4252-b585-1a0d34aeeaa9" (UID: "8d970934-54ed-4252-b585-1a0d34aeeaa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:12.562662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.562635 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.562662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.562657 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.562810 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.562671 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d970934-54ed-4252-b585-1a0d34aeeaa9-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.562810 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.562684 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a020b72-5cdf-4826-800c-c43eb3708ec2-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.562810 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.562696 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.564023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.563987 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-socket" pod "8d970934-54ed-4252-b585-1a0d34aeeaa9" (UID: "8d970934-54ed-4252-b585-1a0d34aeeaa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:12.564131 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.564061 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-socket" pod "9a020b72-5cdf-4826-800c-c43eb3708ec2" (UID: "9a020b72-5cdf-4826-800c-c43eb3708ec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:12.564131 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.564107 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-certs" pod "8d970934-54ed-4252-b585-1a0d34aeeaa9" (UID: "8d970934-54ed-4252-b585-1a0d34aeeaa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:12.564250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.564159 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-certs" pod "9a020b72-5cdf-4826-800c-c43eb3708ec2" (UID: "9a020b72-5cdf-4826-800c-c43eb3708ec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:12.564317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.564284 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-credential-socket" pod "9a020b72-5cdf-4826-800c-c43eb3708ec2" (UID: "9a020b72-5cdf-4826-800c-c43eb3708ec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:12.564506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.564449 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-podinfo" pod "9a020b72-5cdf-4826-800c-c43eb3708ec2" (UID: "9a020b72-5cdf-4826-800c-c43eb3708ec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:11:12.564654 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.564615 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-envoy" pod "8d970934-54ed-4252-b585-1a0d34aeeaa9" (UID: "8d970934-54ed-4252-b585-1a0d34aeeaa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:12.564775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.564682 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-kube-api-access-k5f7m" pod "8d970934-54ed-4252-b585-1a0d34aeeaa9" (UID: "8d970934-54ed-4252-b585-1a0d34aeeaa9"). InnerVolumeSpecName "kube-api-access-k5f7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:12.565315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.565295 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-podinfo" pod "8d970934-54ed-4252-b585-1a0d34aeeaa9" (UID: "8d970934-54ed-4252-b585-1a0d34aeeaa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:11:12.565704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.565667 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-envoy" pod "9a020b72-5cdf-4826-800c-c43eb3708ec2" (UID: "9a020b72-5cdf-4826-800c-c43eb3708ec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:12.565793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.565744 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-kube-api-access-rnqrw" pod "9a020b72-5cdf-4826-800c-c43eb3708ec2" (UID: "9a020b72-5cdf-4826-800c-c43eb3708ec2"). InnerVolumeSpecName "kube-api-access-rnqrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:12.566015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.565992 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-token" pod "9a020b72-5cdf-4826-800c-c43eb3708ec2" (UID: "9a020b72-5cdf-4826-800c-c43eb3708ec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:12.566099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.566043 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-token" pod "8d970934-54ed-4252-b585-1a0d34aeeaa9" (UID: "8d970934-54ed-4252-b585-1a0d34aeeaa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:12.664960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.664914 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.664960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.664951 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.664965 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.664978 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5f7m\" (UniqueName: \"kubernetes.io/projected/8d970934-54ed-4252-b585-1a0d34aeeaa9-kube-api-access-k5f7m\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.664992 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.665005 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.665018 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.665028 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.665040 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.665051 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a020b72-5cdf-4826-800c-c43eb3708ec2-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.665064 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnqrw\" (UniqueName: \"kubernetes.io/projected/9a020b72-5cdf-4826-800c-c43eb3708ec2-kube-api-access-rnqrw\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.665077 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a020b72-5cdf-4826-800c-c43eb3708ec2-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.665146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.665089 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d970934-54ed-4252-b585-1a0d34aeeaa9-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:12.748985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:12.747118 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:13.198856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.198825 4952 generic.go:356] "Generic (PLEG): container finished" podID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerID="f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3" exitCode=0 Oct 29 22:11:13.199064 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.198881 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" Oct 29 22:11:13.199064 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.198889 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" event={"ID":"9a020b72-5cdf-4826-800c-c43eb3708ec2","Type":"ContainerDied","Data":"f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3"} Oct 29 22:11:13.199064 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.198968 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d" event={"ID":"9a020b72-5cdf-4826-800c-c43eb3708ec2","Type":"ContainerDied","Data":"eab6c79ae7ca57bf8eb90c23a397e13099d32cfcd8c7e1803a78c513a9b0375e"} Oct 29 22:11:13.199064 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.198991 4952 scope.go:122] "RemoveContainer" containerID="f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3" Oct 29 22:11:13.200580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.200560 4952 generic.go:356] "Generic (PLEG): container finished" podID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerID="895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368" exitCode=0 Oct 29 22:11:13.200691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.200601 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" Oct 29 22:11:13.201091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.200679 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" event={"ID":"8d970934-54ed-4252-b585-1a0d34aeeaa9","Type":"ContainerDied","Data":"895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368"} Oct 29 22:11:13.201091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.200735 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9" event={"ID":"8d970934-54ed-4252-b585-1a0d34aeeaa9","Type":"ContainerDied","Data":"33254a9d6be0e480a6339ae3e41f761aa662c39a96fa2237830c1c62503e460c"} Oct 29 22:11:13.225962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.225938 4952 scope.go:122] "RemoveContainer" containerID="f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3" Oct 29 22:11:13.226433 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:11:13.226391 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3\": container with ID starting with f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3 not found: ID does not exist" containerID="f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3" Oct 29 22:11:13.226558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.226433 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3"} err="failed to get container status \"f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3\": rpc error: code = NotFound desc = could not find container \"f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3\": container with ID starting with f9bb2a68eec8ce7179863ffc7b63d9e605a759b0df799dfb6f742d772716dde3 not found: ID does not exist" Oct 29 22:11:13.226558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.226457 4952 scope.go:122] "RemoveContainer" containerID="895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368" Oct 29 22:11:13.237056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.237025 4952 scope.go:122] "RemoveContainer" containerID="895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368" Oct 29 22:11:13.237607 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:11:13.237585 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368\": container with ID starting with 895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368 not found: ID does not exist" containerID="895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368" Oct 29 22:11:13.237692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.237616 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368"} err="failed to get container status \"895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368\": rpc error: code = NotFound desc = could not find container \"895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368\": container with ID starting with 895e64f166b0b853bfcc10f9e9e123c277f1f3b5eb746feb87e71114b9f0e368 not found: ID does not exist" Oct 29 22:11:13.266712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.266684 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9"] Oct 29 22:11:13.278429 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.278404 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-99xh9"] Oct 29 22:11:13.309960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.309935 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d"] Oct 29 22:11:13.314674 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:13.314652 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-w5w8d"] Oct 29 22:11:14.647685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:14.647650 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" path="/var/lib/kubelet/pods/8d970934-54ed-4252-b585-1a0d34aeeaa9/volumes" Oct 29 22:11:14.648409 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:14.648387 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" path="/var/lib/kubelet/pods/9a020b72-5cdf-4826-800c-c43eb3708ec2/volumes" Oct 29 22:11:15.220522 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.220492 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw"] Oct 29 22:11:15.221152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221135 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ee351cf3-bba3-4a34-a124-f570b8b43a9d" containerName="maas-api" Oct 29 22:11:15.221152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221151 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee351cf3-bba3-4a34-a124-f570b8b43a9d" containerName="maas-api" Oct 29 22:11:15.221297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221172 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" Oct 29 22:11:15.221297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221178 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" Oct 29 22:11:15.221297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221186 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" Oct 29 22:11:15.221297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221190 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" Oct 29 22:11:15.221297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221203 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8a7008d6-5941-4e8b-b0d3-a576a186c8d0" containerName="maas-api" Oct 29 22:11:15.221297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221208 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7008d6-5941-4e8b-b0d3-a576a186c8d0" containerName="maas-api" Oct 29 22:11:15.221297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221214 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4e08b91f-ac58-4a89-acf2-1c3ad0f886e0" containerName="maas-api" Oct 29 22:11:15.221297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221219 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e08b91f-ac58-4a89-acf2-1c3ad0f886e0" containerName="maas-api" Oct 29 22:11:15.221718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221327 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d970934-54ed-4252-b585-1a0d34aeeaa9" containerName="istio-proxy" Oct 29 22:11:15.221718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221339 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a020b72-5cdf-4826-800c-c43eb3708ec2" containerName="istio-proxy" Oct 29 22:11:15.221718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221349 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7008d6-5941-4e8b-b0d3-a576a186c8d0" containerName="maas-api" Oct 29 22:11:15.221718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221358 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e08b91f-ac58-4a89-acf2-1c3ad0f886e0" containerName="maas-api" Oct 29 22:11:15.221718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.221376 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee351cf3-bba3-4a34-a124-f570b8b43a9d" containerName="maas-api" Oct 29 22:11:15.226979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.226959 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.249985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.249961 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw"] Oct 29 22:11:15.256055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.256031 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-796d71-openshift-default-dockercfg-rrwfs\"" Oct 29 22:11:15.301367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.301333 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-credential-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.301512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.301444 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istiod-ca-cert\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.301512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.301496 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-data\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.301600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.301515 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-token\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.301600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.301541 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-envoy\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.301696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.301673 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-podinfo\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.301798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.301782 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfb7l\" (UniqueName: \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-kube-api-access-hfb7l\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.301875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.301860 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.302033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.302017 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-certs\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.385441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.385388 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:15.403686 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.403659 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istiod-ca-cert\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.403858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.403712 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-data\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.403947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.403882 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-token\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.403947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.403934 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-envoy\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.404058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.403980 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-podinfo\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.404115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.404071 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfb7l\" (UniqueName: \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-kube-api-access-hfb7l\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.404168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.404122 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.404224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.404183 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-data\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.404288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.404266 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-certs\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.404406 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.404390 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-credential-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.404527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.404501 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.404589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.404523 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istiod-ca-cert\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.404589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.404540 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-certs\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.404675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.404659 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-credential-socket\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.406267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.406241 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-envoy\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.406685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.406662 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-podinfo\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.428712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.428542 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-token\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.428712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.428691 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfb7l\" (UniqueName: \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-kube-api-access-hfb7l\") pod \"e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.526592 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:11:15.526509 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:11:15.543325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.543299 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:15.879034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.879006 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw"] Oct 29 22:11:15.879706 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:11:15.879679 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcc1f35_7d6c_45f9_8bdc_87ebfeeacd35.slice/crio-acecac2802e5a634bd98d5997d1bf685d07c47a3e7eedc5a9a8519393f72e2cf WatchSource:0}: Error finding container acecac2802e5a634bd98d5997d1bf685d07c47a3e7eedc5a9a8519393f72e2cf: Status 404 returned error can't find the container with id acecac2802e5a634bd98d5997d1bf685d07c47a3e7eedc5a9a8519393f72e2cf Oct 29 22:11:15.882789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.882747 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:11:15.882891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.882833 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:11:15.882891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:15.882876 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:11:16.080532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.080406 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:16.236646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.236608 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" event={"ID":"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35","Type":"ContainerStarted","Data":"ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d"} Oct 29 22:11:16.236805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.236666 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" event={"ID":"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35","Type":"ContainerStarted","Data":"acecac2802e5a634bd98d5997d1bf685d07c47a3e7eedc5a9a8519393f72e2cf"} Oct 29 22:11:16.241730 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:11:16.241702 4952 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca65c2f_2898_48e2_ac26_727c7672ff6d.slice/crio-conmon-6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:11:16.300398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.300345 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" podStartSLOduration=1.300329585 podStartE2EDuration="1.300329585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:16.294652246 +0000 UTC m=+1741.769240662" watchObservedRunningTime="2026-10-29 22:11:16.300329585 +0000 UTC m=+1741.774918003" Oct 29 22:11:16.490047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.490013 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 22:11:16.528947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.528917 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-envoy\") pod \"c0ca98e1-3645-45e0-a041-66af7c97da96\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " Oct 29 22:11:16.529200 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.529180 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-data\") pod \"c0ca98e1-3645-45e0-a041-66af7c97da96\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " Oct 29 22:11:16.529398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.529372 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-data" pod "c0ca98e1-3645-45e0-a041-66af7c97da96" (UID: "c0ca98e1-3645-45e0-a041-66af7c97da96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:16.529581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.529555 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-credential-socket\") pod \"c0ca98e1-3645-45e0-a041-66af7c97da96\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " Oct 29 22:11:16.529730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.529598 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-certs\") pod \"c0ca98e1-3645-45e0-a041-66af7c97da96\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " Oct 29 22:11:16.529730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.529651 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-socket\") pod \"c0ca98e1-3645-45e0-a041-66af7c97da96\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " Oct 29 22:11:16.529730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.529700 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-podinfo\") pod \"c0ca98e1-3645-45e0-a041-66af7c97da96\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " Oct 29 22:11:16.529922 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.529748 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ca98e1-3645-45e0-a041-66af7c97da96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ca98e1-3645-45e0-a041-66af7c97da96-istiod-ca-cert\") pod \"c0ca98e1-3645-45e0-a041-66af7c97da96\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " Oct 29 22:11:16.529922 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.529845 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-kube-api-access-pjt8b\" (UniqueName: \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-kube-api-access-pjt8b\") pod \"c0ca98e1-3645-45e0-a041-66af7c97da96\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " Oct 29 22:11:16.529922 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.529879 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-token\" (UniqueName: \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-token\") pod \"c0ca98e1-3645-45e0-a041-66af7c97da96\" (UID: \"c0ca98e1-3645-45e0-a041-66af7c97da96\") " Oct 29 22:11:16.530133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.530040 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-certs" pod "c0ca98e1-3645-45e0-a041-66af7c97da96" (UID: "c0ca98e1-3645-45e0-a041-66af7c97da96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:16.530194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.530176 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-credential-socket" pod "c0ca98e1-3645-45e0-a041-66af7c97da96" (UID: "c0ca98e1-3645-45e0-a041-66af7c97da96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:16.530503 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.530478 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-socket" pod "c0ca98e1-3645-45e0-a041-66af7c97da96" (UID: "c0ca98e1-3645-45e0-a041-66af7c97da96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:16.530734 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.530717 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.530811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.530739 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.530811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.530754 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.530811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.530767 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.530811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.530764 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ca98e1-3645-45e0-a041-66af7c97da96-istiod-ca-cert" pod "c0ca98e1-3645-45e0-a041-66af7c97da96" (UID: "c0ca98e1-3645-45e0-a041-66af7c97da96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:16.533467 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.532312 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-envoy" pod "c0ca98e1-3645-45e0-a041-66af7c97da96" (UID: "c0ca98e1-3645-45e0-a041-66af7c97da96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:16.549773 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.544312 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:16.549773 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.546434 4952 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:11:16.549773 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.546506 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 22:11:16.553992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.553953 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-token" pod "c0ca98e1-3645-45e0-a041-66af7c97da96" (UID: "c0ca98e1-3645-45e0-a041-66af7c97da96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:16.558248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.558093 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-podinfo" pod "c0ca98e1-3645-45e0-a041-66af7c97da96" (UID: "c0ca98e1-3645-45e0-a041-66af7c97da96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:11:16.558432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.558403 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-kube-api-access-pjt8b" pod "c0ca98e1-3645-45e0-a041-66af7c97da96" (UID: "c0ca98e1-3645-45e0-a041-66af7c97da96"). InnerVolumeSpecName "kube-api-access-pjt8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:16.587854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.587829 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 22:11:16.631927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.631871 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-socket\") pod \"cca65c2f-2898-48e2-ac26-727c7672ff6d\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " Oct 29 22:11:16.632106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632042 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-envoy\") pod \"cca65c2f-2898-48e2-ac26-727c7672ff6d\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " Oct 29 22:11:16.632106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632078 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca65c2f-2898-48e2-ac26-727c7672ff6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca65c2f-2898-48e2-ac26-727c7672ff6d-istiod-ca-cert\") pod \"cca65c2f-2898-48e2-ac26-727c7672ff6d\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " Oct 29 22:11:16.632224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632143 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-credential-socket\") pod \"cca65c2f-2898-48e2-ac26-727c7672ff6d\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " Oct 29 22:11:16.632224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632181 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-podinfo\") pod \"cca65c2f-2898-48e2-ac26-727c7672ff6d\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " Oct 29 22:11:16.632327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632225 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-socket" pod "cca65c2f-2898-48e2-ac26-727c7672ff6d" (UID: "cca65c2f-2898-48e2-ac26-727c7672ff6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:16.632327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632286 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-certs\") pod \"cca65c2f-2898-48e2-ac26-727c7672ff6d\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " Oct 29 22:11:16.632438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632324 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-kube-api-access-klwvz\" (UniqueName: \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-kube-api-access-klwvz\") pod \"cca65c2f-2898-48e2-ac26-727c7672ff6d\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " Oct 29 22:11:16.632438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632380 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-data\") pod \"cca65c2f-2898-48e2-ac26-727c7672ff6d\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " Oct 29 22:11:16.632541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632452 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-token\" (UniqueName: \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-token\") pod \"cca65c2f-2898-48e2-ac26-727c7672ff6d\" (UID: \"cca65c2f-2898-48e2-ac26-727c7672ff6d\") " Oct 29 22:11:16.632541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632485 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-credential-socket" pod "cca65c2f-2898-48e2-ac26-727c7672ff6d" (UID: "cca65c2f-2898-48e2-ac26-727c7672ff6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:16.632724 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632533 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca65c2f-2898-48e2-ac26-727c7672ff6d-istiod-ca-cert" pod "cca65c2f-2898-48e2-ac26-727c7672ff6d" (UID: "cca65c2f-2898-48e2-ac26-727c7672ff6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:16.633227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.632877 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-certs" pod "cca65c2f-2898-48e2-ac26-727c7672ff6d" (UID: "cca65c2f-2898-48e2-ac26-727c7672ff6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:16.633352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.633301 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-data" pod "cca65c2f-2898-48e2-ac26-727c7672ff6d" (UID: "cca65c2f-2898-48e2-ac26-727c7672ff6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:16.633610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.633449 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.633610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.633478 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca65c2f-2898-48e2-ac26-727c7672ff6d-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.633610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.633493 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.633610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.633507 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.633610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.633521 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ca98e1-3645-45e0-a041-66af7c97da96-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.633610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.633536 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.633610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.633548 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjt8b\" (UniqueName: \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-kube-api-access-pjt8b\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.633610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.633562 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.633610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.633575 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ca98e1-3645-45e0-a041-66af7c97da96-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.634848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.634814 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-envoy" pod "cca65c2f-2898-48e2-ac26-727c7672ff6d" (UID: "cca65c2f-2898-48e2-ac26-727c7672ff6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:16.636024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.635027 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-podinfo" pod "cca65c2f-2898-48e2-ac26-727c7672ff6d" (UID: "cca65c2f-2898-48e2-ac26-727c7672ff6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:11:16.636024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.635285 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-token" pod "cca65c2f-2898-48e2-ac26-727c7672ff6d" (UID: "cca65c2f-2898-48e2-ac26-727c7672ff6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:16.662377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.653029 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-kube-api-access-klwvz" pod "cca65c2f-2898-48e2-ac26-727c7672ff6d" (UID: "cca65c2f-2898-48e2-ac26-727c7672ff6d"). InnerVolumeSpecName "kube-api-access-klwvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:16.738988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.735044 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.738988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.735083 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.738988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.735098 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.738988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.735110 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klwvz\" (UniqueName: \"kubernetes.io/projected/cca65c2f-2898-48e2-ac26-727c7672ff6d-kube-api-access-klwvz\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:16.738988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:16.735125 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca65c2f-2898-48e2-ac26-727c7672ff6d-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:17.259114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.259062 4952 generic.go:356] "Generic (PLEG): container finished" podID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerID="6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934" exitCode=0 Oct 29 22:11:17.259647 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.259138 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" Oct 29 22:11:17.259647 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.259152 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" event={"ID":"cca65c2f-2898-48e2-ac26-727c7672ff6d","Type":"ContainerDied","Data":"6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934"} Oct 29 22:11:17.259647 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.259248 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl" event={"ID":"cca65c2f-2898-48e2-ac26-727c7672ff6d","Type":"ContainerDied","Data":"ea032c51ccc878083c53a86623004be6862611f83d3b80b64192c878b1a0dbbe"} Oct 29 22:11:17.259647 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.259313 4952 scope.go:122] "RemoveContainer" containerID="6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934" Oct 29 22:11:17.261673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.261646 4952 generic.go:356] "Generic (PLEG): container finished" podID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerID="ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575" exitCode=0 Oct 29 22:11:17.262065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.262042 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" event={"ID":"c0ca98e1-3645-45e0-a041-66af7c97da96","Type":"ContainerDied","Data":"ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575"} Oct 29 22:11:17.262188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.262104 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" event={"ID":"c0ca98e1-3645-45e0-a041-66af7c97da96","Type":"ContainerDied","Data":"667188ffebf416801788c13a20bbdceffba55675972c3418d1b0293042614b4b"} Oct 29 22:11:17.262188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.261965 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg" Oct 29 22:11:17.272491 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.272467 4952 scope.go:122] "RemoveContainer" containerID="6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934" Oct 29 22:11:17.272850 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:11:17.272823 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934\": container with ID starting with 6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934 not found: ID does not exist" containerID="6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934" Oct 29 22:11:17.272949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.272861 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934"} err="failed to get container status \"6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934\": rpc error: code = NotFound desc = could not find container \"6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934\": container with ID starting with 6faf4bee3710db4af9435f7786fb8a8bca4a287ea785e445f7653d5edcefb934 not found: ID does not exist" Oct 29 22:11:17.272949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.272885 4952 scope.go:122] "RemoveContainer" containerID="ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575" Oct 29 22:11:17.283387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.283365 4952 scope.go:122] "RemoveContainer" containerID="ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575" Oct 29 22:11:17.283711 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:11:17.283684 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575\": container with ID starting with ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575 not found: ID does not exist" containerID="ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575" Oct 29 22:11:17.283798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.283718 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575"} err="failed to get container status \"ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575\": rpc error: code = NotFound desc = could not find container \"ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575\": container with ID starting with ca912536203b8a5f768841c7bedbf4f0fbc30efe88aaef25c977df16e58b5575 not found: ID does not exist" Oct 29 22:11:17.361868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.361831 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl"] Oct 29 22:11:17.368725 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.368696 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-l29fl"] Oct 29 22:11:17.404437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.404399 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg"] Oct 29 22:11:17.411684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.411655 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fh4qg"] Oct 29 22:11:17.547992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.547465 4952 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:11:17.547992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:17.547556 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 22:11:18.265948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:18.253861 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:18.270949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:18.269860 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4"] Oct 29 22:11:18.270949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:18.270156 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" containerID="cri-o://b6decfa05bdd76780ee0b17ad6b1531ba32ebfce13e326723d4791d4c0aba981" gracePeriod=30 Oct 29 22:11:18.552977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:18.549315 4952 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:11:18.552977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:18.549380 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 22:11:18.597831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:18.593532 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:18.672942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:18.668355 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" path="/var/lib/kubelet/pods/c0ca98e1-3645-45e0-a041-66af7c97da96/volumes" Oct 29 22:11:18.672942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:18.669050 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" path="/var/lib/kubelet/pods/cca65c2f-2898-48e2-ac26-727c7672ff6d/volumes" Oct 29 22:11:19.547936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:19.547275 4952 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:11:19.547936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:19.547335 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 22:11:19.813824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:19.813731 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:20.250095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.250064 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf"] Oct 29 22:11:20.251268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.251240 4952 cpu_manager.go:414] "containerMap: removing container" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerName="istio-proxy" Oct 29 22:11:20.251433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.251417 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerName="istio-proxy" Oct 29 22:11:20.251565 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.251554 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" Oct 29 22:11:20.251669 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.251654 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" Oct 29 22:11:20.251978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.251963 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca65c2f-2898-48e2-ac26-727c7672ff6d" containerName="istio-proxy" Oct 29 22:11:20.252114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.252087 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ca98e1-3645-45e0-a041-66af7c97da96" containerName="istio-proxy" Oct 29 22:11:20.258413 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.258391 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.279449 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.279417 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4c5549-openshift-default-dockercfg-mqp56\"" Oct 29 22:11:20.307196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.307163 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf"] Oct 29 22:11:20.342932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.325833 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edadf27c-b54d-403b-8618-e240e0b9f728-istio-podinfo\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.342932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.325953 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edadf27c-b54d-403b-8618-e240e0b9f728-istiod-ca-cert\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.342932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.325988 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-certs\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.342932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.326040 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-envoy\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.342932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.326116 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frp2z\" (UniqueName: \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-kube-api-access-frp2z\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.342932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.326152 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-data\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.342932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.326193 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-socket\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.342932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.326237 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-istio-token\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.342932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.326268 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-credential-socket\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.443081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.443041 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:20.445972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.443953 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edadf27c-b54d-403b-8618-e240e0b9f728-istio-podinfo\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.486701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.485988 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edadf27c-b54d-403b-8618-e240e0b9f728-istiod-ca-cert\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.486701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.486097 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-certs\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.486701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.486182 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-envoy\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.486701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.486297 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frp2z\" (UniqueName: \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-kube-api-access-frp2z\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.486701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.486344 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-data\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.486701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.486404 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-socket\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.486701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.486459 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-istio-token\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.486701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.486507 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-credential-socket\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.512466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.487209 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-credential-socket\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.512466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.495288 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edadf27c-b54d-403b-8618-e240e0b9f728-istiod-ca-cert\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.558928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.545259 4952 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:11:20.558928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.545321 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 22:11:20.558928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.545881 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-certs\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.558928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.550265 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-socket\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.558928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.550696 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edadf27c-b54d-403b-8618-e240e0b9f728-istio-podinfo\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.614930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.567122 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-data\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.740176 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.724399 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-envoy\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.740176 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.724975 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-istio-token\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.830021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.804326 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:20.873763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.873728 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frp2z\" (UniqueName: \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-kube-api-access-frp2z\") pod \"e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:20.898692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.897288 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:21.000926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:20.998579 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:21.171186 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:21.170481 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:21.394044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:21.394018 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf"] Oct 29 22:11:21.395161 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:11:21.395126 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedadf27c_b54d_403b_8618_e240e0b9f728.slice/crio-7a79fd5fcc86cfbfda6f3b219f5a7d792483a01d62e6f179f29ef8f1cb6c9927 WatchSource:0}: Error finding container 7a79fd5fcc86cfbfda6f3b219f5a7d792483a01d62e6f179f29ef8f1cb6c9927: Status 404 returned error can't find the container with id 7a79fd5fcc86cfbfda6f3b219f5a7d792483a01d62e6f179f29ef8f1cb6c9927 Oct 29 22:11:21.398444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:21.398403 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:11:21.398524 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:21.398491 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:11:21.398579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:21.398534 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:11:21.402879 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:21.402777 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:21.561355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:21.546007 4952 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:11:21.561355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:21.546063 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 22:11:21.594946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:21.593410 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:22.414929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:22.414760 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" event={"ID":"edadf27c-b54d-403b-8618-e240e0b9f728","Type":"ContainerStarted","Data":"e7552e8bcc313659fed883b6f3e682bac8915fa9ab907819bf5ece9f9a1c9ec1"} Oct 29 22:11:22.414929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:22.414829 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" event={"ID":"edadf27c-b54d-403b-8618-e240e0b9f728","Type":"ContainerStarted","Data":"7a79fd5fcc86cfbfda6f3b219f5a7d792483a01d62e6f179f29ef8f1cb6c9927"} Oct 29 22:11:22.493569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:22.493509 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" podStartSLOduration=2.493490045 podStartE2EDuration="2.493490045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:22.486564166 +0000 UTC m=+1747.961152585" watchObservedRunningTime="2026-10-29 22:11:22.493490045 +0000 UTC m=+1747.968078464" Oct 29 22:11:22.546925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:22.546222 4952 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:11:22.546925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:22.546287 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 29 22:11:22.898191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:22.898154 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:22.905092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:22.905066 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:23.141153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.141114 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:23.429339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.429297 4952 generic.go:356] "Generic (PLEG): container finished" podID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerID="b6decfa05bdd76780ee0b17ad6b1531ba32ebfce13e326723d4791d4c0aba981" exitCode=0 Oct 29 22:11:23.430804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.430773 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" event={"ID":"92724a9e-daa1-4839-b12e-38db23b6f8be","Type":"ContainerDied","Data":"b6decfa05bdd76780ee0b17ad6b1531ba32ebfce13e326723d4791d4c0aba981"} Oct 29 22:11:23.430957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.430843 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:23.432754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.432481 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:11:23.560841 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.560513 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:23.563472 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.562150 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:23.566949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.564270 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:11:23.645223 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.645186 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 22:11:23.783478 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.783443 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-credential-socket\") pod \"92724a9e-daa1-4839-b12e-38db23b6f8be\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " Oct 29 22:11:23.783658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.783512 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-envoy\") pod \"92724a9e-daa1-4839-b12e-38db23b6f8be\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " Oct 29 22:11:23.783658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.783566 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92724a9e-daa1-4839-b12e-38db23b6f8be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92724a9e-daa1-4839-b12e-38db23b6f8be-istiod-ca-cert\") pod \"92724a9e-daa1-4839-b12e-38db23b6f8be\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " Oct 29 22:11:23.783658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.783600 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-token\" (UniqueName: \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-token\") pod \"92724a9e-daa1-4839-b12e-38db23b6f8be\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " Oct 29 22:11:23.783858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.783664 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-podinfo\") pod \"92724a9e-daa1-4839-b12e-38db23b6f8be\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " Oct 29 22:11:23.783858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.783730 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-socket\") pod \"92724a9e-daa1-4839-b12e-38db23b6f8be\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " Oct 29 22:11:23.783858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.783754 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-data\") pod \"92724a9e-daa1-4839-b12e-38db23b6f8be\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " Oct 29 22:11:23.783858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.783819 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-kube-api-access-l5twm\" (UniqueName: \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-kube-api-access-l5twm\") pod \"92724a9e-daa1-4839-b12e-38db23b6f8be\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " Oct 29 22:11:23.784142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.783861 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-certs\") pod \"92724a9e-daa1-4839-b12e-38db23b6f8be\" (UID: \"92724a9e-daa1-4839-b12e-38db23b6f8be\") " Oct 29 22:11:23.784496 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.784471 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-certs" pod "92724a9e-daa1-4839-b12e-38db23b6f8be" (UID: "92724a9e-daa1-4839-b12e-38db23b6f8be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:23.784672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.784655 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-credential-socket" pod "92724a9e-daa1-4839-b12e-38db23b6f8be" (UID: "92724a9e-daa1-4839-b12e-38db23b6f8be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:23.787322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.787288 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-envoy" pod "92724a9e-daa1-4839-b12e-38db23b6f8be" (UID: "92724a9e-daa1-4839-b12e-38db23b6f8be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:23.787796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.787529 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-socket" pod "92724a9e-daa1-4839-b12e-38db23b6f8be" (UID: "92724a9e-daa1-4839-b12e-38db23b6f8be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:23.788283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.788241 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-data" pod "92724a9e-daa1-4839-b12e-38db23b6f8be" (UID: "92724a9e-daa1-4839-b12e-38db23b6f8be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:11:23.788624 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.788602 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92724a9e-daa1-4839-b12e-38db23b6f8be-istiod-ca-cert" pod "92724a9e-daa1-4839-b12e-38db23b6f8be" (UID: "92724a9e-daa1-4839-b12e-38db23b6f8be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:11:23.790067 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.790041 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-podinfo" pod "92724a9e-daa1-4839-b12e-38db23b6f8be" (UID: "92724a9e-daa1-4839-b12e-38db23b6f8be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:11:23.790240 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.790217 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-token" pod "92724a9e-daa1-4839-b12e-38db23b6f8be" (UID: "92724a9e-daa1-4839-b12e-38db23b6f8be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:23.793254 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.793216 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-kube-api-access-l5twm" pod "92724a9e-daa1-4839-b12e-38db23b6f8be" (UID: "92724a9e-daa1-4839-b12e-38db23b6f8be"). InnerVolumeSpecName "kube-api-access-l5twm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:11:23.889309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.886245 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5twm\" (UniqueName: \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-kube-api-access-l5twm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:23.889309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.886281 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:23.889309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.886296 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:23.889309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.886310 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:23.889309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.886324 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92724a9e-daa1-4839-b12e-38db23b6f8be-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:23.889309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.886338 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:23.889309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.886351 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:23.889309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.886364 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:23.889309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:23.886377 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92724a9e-daa1-4839-b12e-38db23b6f8be-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:11:24.197164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:24.196951 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:11:24.443670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:24.443641 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" Oct 29 22:11:24.444135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:24.444101 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4" event={"ID":"92724a9e-daa1-4839-b12e-38db23b6f8be","Type":"ContainerDied","Data":"4744e8b57b69c22c13ab431f786d538bda891f3519eba13639e623bf0ad5d76c"} Oct 29 22:11:24.444308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:24.444296 4952 scope.go:122] "RemoveContainer" containerID="b6decfa05bdd76780ee0b17ad6b1531ba32ebfce13e326723d4791d4c0aba981" Oct 29 22:11:24.516985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:24.516898 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4"] Oct 29 22:11:24.523544 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:24.523516 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-9skd4"] Oct 29 22:11:24.652065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:24.652028 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" path="/var/lib/kubelet/pods/92724a9e-daa1-4839-b12e-38db23b6f8be/volumes" Oct 29 22:11:25.020495 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:25.020462 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:25.193568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:25.193530 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:25.368840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:25.368803 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:27.165264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.113609 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.207944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.194171 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58"] Oct 29 22:11:27.207944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.195174 4952 cpu_manager.go:414] "containerMap: removing container" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" Oct 29 22:11:27.207944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.195196 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" Oct 29 22:11:27.207944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.195376 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="92724a9e-daa1-4839-b12e-38db23b6f8be" containerName="istio-proxy" Oct 29 22:11:27.207944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.201277 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:11:27.246944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.208735 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:27.246944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.237009 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.322930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.254857 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58\" (UID: \"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:11:27.322930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.254934 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs94g\" (UniqueName: \"kubernetes.io/projected/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-kube-api-access-vs94g\") pod \"payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58\" (UID: \"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:11:27.322930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.296861 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-05687d\"" Oct 29 22:11:27.322930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.297636 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-05687d-dockercfg-flhdf\"" Oct 29 22:11:27.323449 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.323410 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.324380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.324347 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.447984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.440153 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58\" (UID: \"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:11:27.447984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.440223 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs94g\" (UniqueName: \"kubernetes.io/projected/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-kube-api-access-vs94g\") pod \"payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58\" (UID: \"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:11:27.562231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.485048 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.562231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.511978 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58\" (UID: \"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:11:27.600939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.586364 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58"] Oct 29 22:11:27.612964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.604120 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:11:27.613331 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.613288 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 22:11:27.641951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.622190 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:27.672326 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.672285 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:11:27.720013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.705305 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.924936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.923386 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:27.924936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.923822 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 22:11:27.924936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:27.924179 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 22:11:28.184848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.184813 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs94g\" (UniqueName: \"kubernetes.io/projected/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-kube-api-access-vs94g\") pod \"payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58\" (UID: \"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:11:28.237735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.237672 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.250918 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.250870 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:11:28.279073 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.279036 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj"] Oct 29 22:11:28.326136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.326094 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.333122 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.333080 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 29 22:11:28.354058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.354023 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:11:28.468974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.468866 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:11:28.469227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.469209 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 22:11:28.486016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.485983 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:11:28.490011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.489971 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.510780 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.509734 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 22:11:28.559993 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.555822 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.607455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.607408 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.742548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.611593 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 22:11:28.760930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.749757 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f973a23-7f51-4459-b4db-c125fb274aae-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj\" (UID: \"7f973a23-7f51-4459-b4db-c125fb274aae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:11:28.760930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.749945 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhfg\" (UniqueName: \"kubernetes.io/projected/7f973a23-7f51-4459-b4db-c125fb274aae-kube-api-access-hmhfg\") pod \"payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj\" (UID: \"7f973a23-7f51-4459-b4db-c125fb274aae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:11:28.760930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.677329 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.760930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.714770 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.773931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.614089 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:28.860930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.835884 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj"] Oct 29 22:11:28.860930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.836012 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:11:28.860930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.836050 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:11:28.860930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.836087 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:11:28.860930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.836121 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 22:11:28.860930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.836156 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 22:11:28.860930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.836186 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 22:11:28.975926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.975073 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 22:11:28.999985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:28.989252 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:11:29.003553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.003520 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 22:11:29.003808 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.003791 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 22:11:29.025087 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.025048 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhfg\" (UniqueName: \"kubernetes.io/projected/7f973a23-7f51-4459-b4db-c125fb274aae-kube-api-access-hmhfg\") pod \"payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj\" (UID: \"7f973a23-7f51-4459-b4db-c125fb274aae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:11:29.030837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.025546 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f973a23-7f51-4459-b4db-c125fb274aae-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj\" (UID: \"7f973a23-7f51-4459-b4db-c125fb274aae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:11:29.068650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.033580 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f973a23-7f51-4459-b4db-c125fb274aae-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj\" (UID: \"7f973a23-7f51-4459-b4db-c125fb274aae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:11:29.095396 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.095325 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhfg\" (UniqueName: \"kubernetes.io/projected/7f973a23-7f51-4459-b4db-c125fb274aae-kube-api-access-hmhfg\") pod \"payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj\" (UID: \"7f973a23-7f51-4459-b4db-c125fb274aae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:11:29.141569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.141537 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 22:11:29.203602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.203547 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:29.286628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.286604 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:11:29.333036 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.330877 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 22:11:29.341546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.340363 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 22:11:29.341546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.341021 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 22:11:29.346545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.346510 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:11:29.568412 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.568386 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58"] Oct 29 22:11:29.672819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:29.672781 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj"] Oct 29 22:11:30.049060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.049022 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:30.182140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.182109 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk"] Oct 29 22:11:30.186617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.186593 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:11:30.199713 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.199689 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqq7r\" (UniqueName: \"kubernetes.io/projected/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-kube-api-access-mqq7r\") pod \"payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk\" (UID: \"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:11:30.199854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.199832 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk\" (UID: \"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:11:30.205360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.205276 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4c5549\"" Oct 29 22:11:30.207006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.206959 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4c5549-dockercfg-lsggv\"" Oct 29 22:11:30.214322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.214294 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk"] Oct 29 22:11:30.223514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.223366 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:30.228315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.228280 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" event={"ID":"7f973a23-7f51-4459-b4db-c125fb274aae","Type":"ContainerStarted","Data":"1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28"} Oct 29 22:11:30.228438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.228346 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" event={"ID":"7f973a23-7f51-4459-b4db-c125fb274aae","Type":"ContainerStarted","Data":"5fa855222b5618f33f63e52ec77c2bae51bba37b32c95abc173abfa10e28051b"} Oct 29 22:11:30.228438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.228400 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:11:30.230924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.230888 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" event={"ID":"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4","Type":"ContainerStarted","Data":"06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944"} Oct 29 22:11:30.231104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.231081 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" event={"ID":"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4","Type":"ContainerStarted","Data":"bcc21ec867935b97682928c7f2219ac921a74c377631cd9e05b1007b6caa0fc9"} Oct 29 22:11:30.231827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.231805 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:11:30.409403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.409318 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" podStartSLOduration=3.40929331 podStartE2EDuration="3.40929331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:30.366234217 +0000 UTC m=+1755.840822687" watchObservedRunningTime="2026-10-29 22:11:30.40929331 +0000 UTC m=+1755.883881729" Oct 29 22:11:30.459868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.459831 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqq7r\" (UniqueName: \"kubernetes.io/projected/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-kube-api-access-mqq7r\") pod \"payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk\" (UID: \"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:11:30.461927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.461880 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk\" (UID: \"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:11:30.463152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.463127 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk\" (UID: \"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:11:30.463315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.460356 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" podStartSLOduration=3.460337959 podStartE2EDuration="3.460337959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:30.450803462 +0000 UTC m=+1755.925391881" watchObservedRunningTime="2026-10-29 22:11:30.460337959 +0000 UTC m=+1755.934926380" Oct 29 22:11:30.520980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.520755 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqq7r\" (UniqueName: \"kubernetes.io/projected/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-kube-api-access-mqq7r\") pod \"payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk\" (UID: \"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:11:30.529344 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.529310 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:11:30.915508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:30.915478 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk"] Oct 29 22:11:30.917540 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:11:30.917508 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd5575c_0897_4cd9_8b95_dfb6cf7e7cc1.slice/crio-376765cc3fa675580c813f00f96c2e40c5c33d26304989765846c29844038a83 WatchSource:0}: Error finding container 376765cc3fa675580c813f00f96c2e40c5c33d26304989765846c29844038a83: Status 404 returned error can't find the container with id 376765cc3fa675580c813f00f96c2e40c5c33d26304989765846c29844038a83 Oct 29 22:11:31.244678 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:31.244589 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" event={"ID":"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1","Type":"ContainerStarted","Data":"46344336d75f4c95b09568a5021d933ddf6a7faad0ab564c9662bab6892fdf6d"} Oct 29 22:11:31.244678 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:31.244644 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" event={"ID":"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1","Type":"ContainerStarted","Data":"376765cc3fa675580c813f00f96c2e40c5c33d26304989765846c29844038a83"} Oct 29 22:11:31.245135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:31.244939 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:11:31.291472 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:31.291413 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" podStartSLOduration=1.29139935 podStartE2EDuration="1.29139935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:31.289552126 +0000 UTC m=+1756.764140544" watchObservedRunningTime="2026-10-29 22:11:31.29139935 +0000 UTC m=+1756.765987784" Oct 29 22:11:31.552400 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:31.552301 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:33.266306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:33.261593 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:34.931460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:34.931363 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:36.649236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:36.649196 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:38.339534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:38.339490 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:40.154771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:40.154725 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:41.249163 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.249130 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:11:41.274740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.274706 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:41.649756 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.649716 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd"] Oct 29 22:11:41.654673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.654642 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.681817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.681782 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd"] Oct 29 22:11:41.686759 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.686727 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-99e420-openshift-default-dockercfg-7pkqn\"" Oct 29 22:11:41.769575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.769535 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:41.798931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.797818 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-podinfo\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.798931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.797930 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-credential-socket\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.798931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.797960 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-token\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.798931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.798005 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-data\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.798931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.798163 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lmx9\" (UniqueName: \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-kube-api-access-6lmx9\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.798931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.798276 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istiod-ca-cert\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.798931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.798311 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-certs\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.798931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.798399 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-envoy\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.798931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.798465 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-socket\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.916972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.916811 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-envoy\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.917741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.917712 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-socket\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.918344 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.918321 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-podinfo\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.918523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.918505 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-credential-socket\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.918641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.918626 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-token\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.918779 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.918764 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-data\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.919119 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.919098 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lmx9\" (UniqueName: \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-kube-api-access-6lmx9\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.919322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.919305 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istiod-ca-cert\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.919442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.919428 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-certs\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.919888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.919867 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-certs\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.920097 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.918227 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-socket\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.920829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.920806 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-data\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.921226 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.921205 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-credential-socket\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.922506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.922484 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istiod-ca-cert\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.942515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.939604 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-podinfo\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.942515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.941331 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-envoy\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.985952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.964402 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lmx9\" (UniqueName: \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-kube-api-access-6lmx9\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:41.985952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:41.969215 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-token\") pod \"e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:42.000243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:42.000211 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:42.001497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:42.001442 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:42.222092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:42.209734 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:42.256505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:42.256293 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:11:42.258939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:42.258724 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:11:42.400263 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:42.400233 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd"] Oct 29 22:11:42.404772 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:42.404732 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:11:42.404900 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:42.404823 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:11:42.404900 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:42.404867 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:11:43.380130 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:43.380087 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" event={"ID":"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1","Type":"ContainerStarted","Data":"89c48aa2572127a93362586fe8b79e2589467ae0dd691ceecae21b7997b6cae6"} Oct 29 22:11:43.380543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:43.380156 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" event={"ID":"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1","Type":"ContainerStarted","Data":"402c70345fef1a6c70c1477a85a8cbb42a80f878fe27163d99b1d3e736cf8475"} Oct 29 22:11:43.442151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:43.442098 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" podStartSLOduration=2.442082417 podStartE2EDuration="2.442082417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:43.437998748 +0000 UTC m=+1768.912587167" watchObservedRunningTime="2026-10-29 22:11:43.442082417 +0000 UTC m=+1768.916670835" Oct 29 22:11:43.752873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:43.746941 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:44.001435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:44.001389 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:44.007482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:44.007412 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:44.388392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:44.388364 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:44.389453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:44.389430 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:11:45.489016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:45.488979 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:45.547012 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:11:45.546400 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:11:45.876746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:45.876711 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:46.061143 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:46.061094 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:47.643984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:47.643946 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:47.827487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:47.827364 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:48.021559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:48.021458 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:48.674792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:48.674732 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:49.437931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:49.435760 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:49.744540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:49.744452 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:49.928665 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:49.928571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:51.770627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:51.770588 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:51.947054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:51.947016 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:52.394105 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:52.394062 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:53.328824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.328786 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696"] Oct 29 22:11:53.336395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.336361 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.356535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.356502 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696"] Oct 29 22:11:53.386609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.386573 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01abd238-8ef8-4e3e-9503-49304f28490b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.386806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.386657 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-metrics-certs\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.386806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.386775 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb88r\" (UniqueName: \"kubernetes.io/projected/01abd238-8ef8-4e3e-9503-49304f28490b-kube-api-access-bb88r\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.386978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.386806 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.394566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.394533 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4c5549-serving-cert\"" Oct 29 22:11:53.473187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.473146 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:53.488533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.488499 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb88r\" (UniqueName: \"kubernetes.io/projected/01abd238-8ef8-4e3e-9503-49304f28490b-kube-api-access-bb88r\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.488705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.488546 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.488705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.488694 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01abd238-8ef8-4e3e-9503-49304f28490b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.488818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.488739 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-metrics-certs\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.489402 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.489377 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01abd238-8ef8-4e3e-9503-49304f28490b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.491109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.491088 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-metrics-certs\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.491215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.491202 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.512885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.512853 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb88r\" (UniqueName: \"kubernetes.io/projected/01abd238-8ef8-4e3e-9503-49304f28490b-kube-api-access-bb88r\") pod \"maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:53.657072 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:53.656983 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:54.041552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:54.041523 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696"] Oct 29 22:11:54.043195 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:11:54.043163 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01abd238_8ef8_4e3e_9503_49304f28490b.slice/crio-7430bdfc54af0f7800bcde9ea96a0740abec4885751bd069cbf8b309041f5e23 WatchSource:0}: Error finding container 7430bdfc54af0f7800bcde9ea96a0740abec4885751bd069cbf8b309041f5e23: Status 404 returned error can't find the container with id 7430bdfc54af0f7800bcde9ea96a0740abec4885751bd069cbf8b309041f5e23 Oct 29 22:11:54.044554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:54.044533 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:11:54.621065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:54.621025 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" event={"ID":"01abd238-8ef8-4e3e-9503-49304f28490b","Type":"ContainerStarted","Data":"7430bdfc54af0f7800bcde9ea96a0740abec4885751bd069cbf8b309041f5e23"} Oct 29 22:11:55.396753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:55.396711 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:55.633374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:55.633335 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" event={"ID":"01abd238-8ef8-4e3e-9503-49304f28490b","Type":"ContainerStarted","Data":"7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1"} Oct 29 22:11:55.633872 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:55.633456 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:11:55.693285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:55.691957 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:55.704327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:55.704272 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" podStartSLOduration=2.284602283 podStartE2EDuration="2.704257005s" totalImagesPullingTime="419.654722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:55.692344037 +0000 UTC m=+1781.166932455" watchObservedRunningTime="2026-10-29 22:11:55.704257005 +0000 UTC m=+1781.178845480" Oct 29 22:11:56.117564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:56.117521 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:56.288475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:56.288438 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:56.558771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:56.557916 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:57.583932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:57.581694 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:11:59.316714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:11:59.316594 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:01.079631 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:01.079587 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:01.722098 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:01.722064 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:12:02.939633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:02.939587 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:05.005238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:05.005190 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:05.174335 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:05.174302 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:07.126571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:07.126529 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:08.568369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:08.568320 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:08.877923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:08.877294 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq"] Oct 29 22:12:08.877923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:08.877536 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" podUID="79e59696-ddbd-4c04-8fc3-9e0796650c2a" containerName="maas-api" containerID="cri-o://421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c" gracePeriod=30 Oct 29 22:12:09.062274 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.050589 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:09.244088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.224606 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:09.483952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.483127 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:09.808721 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.808691 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:12:09.818007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.817976 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:09.888994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.888949 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e59696-ddbd-4c04-8fc3-9e0796650c2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e59696-ddbd-4c04-8fc3-9e0796650c2a-openshift-service-ca\") pod \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " Oct 29 22:12:09.889202 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.889000 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e59696-ddbd-4c04-8fc3-9e0796650c2a-kube-api-access-nfk8m\" (UniqueName: \"kubernetes.io/projected/79e59696-ddbd-4c04-8fc3-9e0796650c2a-kube-api-access-nfk8m\") pod \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " Oct 29 22:12:09.889202 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.889049 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-maas-api-tls\") pod \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " Oct 29 22:12:09.889401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.889368 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e59696-ddbd-4c04-8fc3-9e0796650c2a-openshift-service-ca" pod "79e59696-ddbd-4c04-8fc3-9e0796650c2a" (UID: "79e59696-ddbd-4c04-8fc3-9e0796650c2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:09.889583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.889431 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-metrics-certs\") pod \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\" (UID: \"79e59696-ddbd-4c04-8fc3-9e0796650c2a\") " Oct 29 22:12:09.890339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.890313 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e59696-ddbd-4c04-8fc3-9e0796650c2a-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:09.893743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.893701 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-metrics-certs" pod "79e59696-ddbd-4c04-8fc3-9e0796650c2a" (UID: "79e59696-ddbd-4c04-8fc3-9e0796650c2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:09.893887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.893863 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e59696-ddbd-4c04-8fc3-9e0796650c2a-kube-api-access-nfk8m" pod "79e59696-ddbd-4c04-8fc3-9e0796650c2a" (UID: "79e59696-ddbd-4c04-8fc3-9e0796650c2a"). InnerVolumeSpecName "kube-api-access-nfk8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:09.895025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.895000 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-maas-api-tls" pod "79e59696-ddbd-4c04-8fc3-9e0796650c2a" (UID: "79e59696-ddbd-4c04-8fc3-9e0796650c2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:12:09.908820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.908791 4952 generic.go:356] "Generic (PLEG): container finished" podID="79e59696-ddbd-4c04-8fc3-9e0796650c2a" containerID="421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c" exitCode=0 Oct 29 22:12:09.909003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.908980 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" event={"ID":"79e59696-ddbd-4c04-8fc3-9e0796650c2a","Type":"ContainerDied","Data":"421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c"} Oct 29 22:12:09.909077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.909041 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" event={"ID":"79e59696-ddbd-4c04-8fc3-9e0796650c2a","Type":"ContainerDied","Data":"2217c4344a1e4d0cdce37ba0e7fa5bfd28bc25a01d8a5e010b413a3fbefa99ce"} Oct 29 22:12:09.909077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.909073 4952 scope.go:122] "RemoveContainer" containerID="421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c" Oct 29 22:12:09.909265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.909249 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq" Oct 29 22:12:09.934874 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.934845 4952 scope.go:122] "RemoveContainer" containerID="421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c" Oct 29 22:12:09.935451 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:12:09.935424 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c\": container with ID starting with 421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c not found: ID does not exist" containerID="421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c" Oct 29 22:12:09.935556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.935460 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c"} err="failed to get container status \"421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c\": rpc error: code = NotFound desc = could not find container \"421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c\": container with ID starting with 421bd5e26d2cb7575ac70e6962781cb922bc5c25f68325e84b45cd605789c33c not found: ID does not exist" Oct 29 22:12:09.982020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.981866 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq"] Oct 29 22:12:09.987551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.987508 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-796d71-5c86c657d-7glwq"] Oct 29 22:12:09.988301 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.988270 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:09.992299 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.992278 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:09.992380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.992317 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfk8m\" (UniqueName: \"kubernetes.io/projected/79e59696-ddbd-4c04-8fc3-9e0796650c2a-kube-api-access-nfk8m\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:09.992380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:09.992327 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e59696-ddbd-4c04-8fc3-9e0796650c2a-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:10.160844 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:10.160783 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:10.578380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:10.578327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:10.647925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:10.647587 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e59696-ddbd-4c04-8fc3-9e0796650c2a" path="/var/lib/kubelet/pods/79e59696-ddbd-4c04-8fc3-9e0796650c2a/volumes" Oct 29 22:12:11.957969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:11.957083 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:13.680233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:13.680181 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:13.846336 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:13.846289 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:15.218765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:15.218718 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:15.288875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:15.288839 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:12:15.289893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:15.289871 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:12:15.446633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:15.446597 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:15.551201 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:12:15.551160 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:12:15.611895 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:15.611849 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:15.844644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:15.833069 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:16.021557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:16.021514 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:16.542375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:16.542332 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:17.580249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:17.580215 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:17.755029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:17.754990 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:17.936997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:17.936892 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:18.114083 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:18.114035 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:19.826350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:19.826312 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:21.784544 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:21.784511 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:23.540090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:23.540057 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:23.719055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:23.719015 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:23.893871 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:23.893836 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:24.117523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:24.117483 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:24.527015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:24.526973 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:25.627021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:25.626895 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:26.236194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:26.236155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:26.419320 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:26.419286 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:26.721600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:26.721564 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:27.268049 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:27.268008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:27.460984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:27.460848 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:29.039003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:29.037794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:29.214796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:29.214762 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:29.394550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:29.394515 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:31.138952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:31.134101 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:31.305377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:31.305345 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:31.529896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:31.529547 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:32.854899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:32.854857 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:33.029997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:33.029955 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:33.374181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:33.374143 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:34.574956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:34.574874 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:34.757888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:34.757847 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:35.092482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:35.092443 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:36.768424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:36.768388 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:38.421973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:38.421936 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:39.264921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:39.264863 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:39.442971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:39.442933 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:39.539003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:39.538868 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:12:40.287581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:40.287539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:40.461923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:40.461282 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:40.621941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:40.621132 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x"] Oct 29 22:12:40.621941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:40.621383 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" containerID="cri-o://9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296" gracePeriod=30 Oct 29 22:12:41.517401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:41.517363 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:42.000004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:41.999356 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:42.706349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:42.706308 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:42.917120 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:42.917078 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:43.372613 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:43.372571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:43.546497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:43.546458 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:43.732986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:43.722425 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:44.110299 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:44.110257 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:44.573304 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:44.573196 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:45.128927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:45.128102 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:45.470680 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:45.470588 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:45.607939 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:12:45.606921 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:12:45.983088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:45.982989 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:46.230932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.229118 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 22:12:46.355023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.343326 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-kube-api-access-xwdd4\" (UniqueName: \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-kube-api-access-xwdd4\") pod \"9d898918-081a-4972-9e4b-2369b33d744e\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " Oct 29 22:12:46.355023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.343559 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d898918-081a-4972-9e4b-2369b33d744e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d898918-081a-4972-9e4b-2369b33d744e-istio-podinfo\") pod \"9d898918-081a-4972-9e4b-2369b33d744e\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " Oct 29 22:12:46.355023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.343610 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-certs\") pod \"9d898918-081a-4972-9e4b-2369b33d744e\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " Oct 29 22:12:46.355023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.343656 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-data\") pod \"9d898918-081a-4972-9e4b-2369b33d744e\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " Oct 29 22:12:46.355023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.343682 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-istio-token\" (UniqueName: \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-istio-token\") pod \"9d898918-081a-4972-9e4b-2369b33d744e\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " Oct 29 22:12:46.355023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.343717 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-envoy\") pod \"9d898918-081a-4972-9e4b-2369b33d744e\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " Oct 29 22:12:46.355023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.343849 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-socket\") pod \"9d898918-081a-4972-9e4b-2369b33d744e\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " Oct 29 22:12:46.355023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.343881 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-credential-socket\") pod \"9d898918-081a-4972-9e4b-2369b33d744e\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " Oct 29 22:12:46.355023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.343924 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d898918-081a-4972-9e4b-2369b33d744e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d898918-081a-4972-9e4b-2369b33d744e-istiod-ca-cert\") pod \"9d898918-081a-4972-9e4b-2369b33d744e\" (UID: \"9d898918-081a-4972-9e4b-2369b33d744e\") " Oct 29 22:12:46.355023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.352109 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d898918-081a-4972-9e4b-2369b33d744e-istiod-ca-cert" pod "9d898918-081a-4972-9e4b-2369b33d744e" (UID: "9d898918-081a-4972-9e4b-2369b33d744e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:46.367745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.367538 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-data" pod "9d898918-081a-4972-9e4b-2369b33d744e" (UID: "9d898918-081a-4972-9e4b-2369b33d744e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:46.368029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.367986 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-credential-socket" pod "9d898918-081a-4972-9e4b-2369b33d744e" (UID: "9d898918-081a-4972-9e4b-2369b33d744e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:46.378073 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.377989 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-socket" pod "9d898918-081a-4972-9e4b-2369b33d744e" (UID: "9d898918-081a-4972-9e4b-2369b33d744e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:46.378740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.378702 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-certs" pod "9d898918-081a-4972-9e4b-2369b33d744e" (UID: "9d898918-081a-4972-9e4b-2369b33d744e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:46.379597 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.379566 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.379758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.379730 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.379849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.379838 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.380001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.379987 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.380098 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.380086 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d898918-081a-4972-9e4b-2369b33d744e-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.387058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.387008 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-kube-api-access-xwdd4" pod "9d898918-081a-4972-9e4b-2369b33d744e" (UID: "9d898918-081a-4972-9e4b-2369b33d744e"). InnerVolumeSpecName "kube-api-access-xwdd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:46.418141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.407568 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-istio-token" pod "9d898918-081a-4972-9e4b-2369b33d744e" (UID: "9d898918-081a-4972-9e4b-2369b33d744e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:46.418141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.409936 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d898918-081a-4972-9e4b-2369b33d744e-istio-podinfo" pod "9d898918-081a-4972-9e4b-2369b33d744e" (UID: "9d898918-081a-4972-9e4b-2369b33d744e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:12:46.418141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.410584 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-envoy" pod "9d898918-081a-4972-9e4b-2369b33d744e" (UID: "9d898918-081a-4972-9e4b-2369b33d744e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:46.441156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.427743 4952 generic.go:356] "Generic (PLEG): container finished" podID="9d898918-081a-4972-9e4b-2369b33d744e" containerID="9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296" exitCode=0 Oct 29 22:12:46.441156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.427887 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" event={"ID":"9d898918-081a-4972-9e4b-2369b33d744e","Type":"ContainerDied","Data":"9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296"} Oct 29 22:12:46.441156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.427954 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" event={"ID":"9d898918-081a-4972-9e4b-2369b33d744e","Type":"ContainerDied","Data":"c0c314638616772d17f5c3a8d98770ef654d318b14cb58fdda4d2b8982aca398"} Oct 29 22:12:46.441156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.427985 4952 scope.go:122] "RemoveContainer" containerID="9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296" Oct 29 22:12:46.441156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.428153 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x" Oct 29 22:12:46.496454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.496428 4952 scope.go:122] "RemoveContainer" containerID="9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296" Oct 29 22:12:46.498764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.498737 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d898918-081a-4972-9e4b-2369b33d744e-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.498764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.498764 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.498988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.498778 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d898918-081a-4972-9e4b-2369b33d744e-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.498988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.498792 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwdd4\" (UniqueName: \"kubernetes.io/projected/9d898918-081a-4972-9e4b-2369b33d744e-kube-api-access-xwdd4\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:46.499097 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:12:46.499067 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296\": container with ID starting with 9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296 not found: ID does not exist" containerID="9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296" Oct 29 22:12:46.499135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.499113 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296"} err="failed to get container status \"9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296\": rpc error: code = NotFound desc = could not find container \"9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296\": container with ID starting with 9f53fc752c21f0e92077199b0cb545af88372ea2a0d59efccbf96e52b9510296 not found: ID does not exist" Oct 29 22:12:46.516565 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.516538 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x"] Oct 29 22:12:46.521689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.521656 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-dlq9x"] Oct 29 22:12:46.673282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:46.671694 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d898918-081a-4972-9e4b-2369b33d744e" path="/var/lib/kubelet/pods/9d898918-081a-4972-9e4b-2369b33d744e/volumes" Oct 29 22:12:47.266928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:47.256463 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:47.732466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:47.732428 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:48.175532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:48.173956 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:48.520474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:48.520383 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:48.742477 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:48.742431 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:49.126927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:49.119919 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:49.289247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:49.289205 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:49.474030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:49.473337 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:49.646510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:49.646431 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:49.719011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:49.718976 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw"] Oct 29 22:12:49.726999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:49.726880 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" containerID="cri-o://ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d" gracePeriod=30 Oct 29 22:12:49.860053 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:49.860018 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:50.493368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:50.493330 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:50.662265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:50.662228 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:51.636609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.636574 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc"] Oct 29 22:12:51.638897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.638869 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" Oct 29 22:12:51.639093 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.639077 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" Oct 29 22:12:51.639193 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.639182 4952 cpu_manager.go:414] "containerMap: removing container" podUID="79e59696-ddbd-4c04-8fc3-9e0796650c2a" containerName="maas-api" Oct 29 22:12:51.641086 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.641067 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e59696-ddbd-4c04-8fc3-9e0796650c2a" containerName="maas-api" Oct 29 22:12:51.641644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.641624 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d898918-081a-4972-9e4b-2369b33d744e" containerName="istio-proxy" Oct 29 22:12:51.641774 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.641763 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e59696-ddbd-4c04-8fc3-9e0796650c2a" containerName="maas-api" Oct 29 22:12:51.670337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.670303 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.681456 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.681422 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc"] Oct 29 22:12:51.743583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.743532 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-05687d-openshift-default-dockercfg-l7t95\"" Oct 29 22:12:51.808391 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.808336 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.808626 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.808462 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.808626 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.808608 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.808786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.808713 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.808786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.808752 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzzlz\" (UniqueName: \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-kube-api-access-vzzlz\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.808900 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.808786 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.809006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.808984 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.809237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.809210 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/938f28fc-c391-46db-aa26-cb709c6eb9ec-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.809316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.809295 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.910354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910264 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.910354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910320 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.910589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910381 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.910589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910442 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.910589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910472 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzzlz\" (UniqueName: \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-kube-api-access-vzzlz\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.910589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910507 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.910589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910541 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.910838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910603 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/938f28fc-c391-46db-aa26-cb709c6eb9ec-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.910838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910666 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.910838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910715 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.911021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910966 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.911021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910991 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.911021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.910966 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.911398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.911379 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/938f28fc-c391-46db-aa26-cb709c6eb9ec-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.912780 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.912762 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.913351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.913334 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.936102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.936059 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzzlz\" (UniqueName: \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-kube-api-access-vzzlz\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:51.936545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:51.936521 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:52.000682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.000638 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:52.369595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.369553 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc"] Oct 29 22:12:52.371299 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:12:52.371259 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938f28fc_c391_46db_aa26_cb709c6eb9ec.slice/crio-3a34c96c843c4eab818c7f0d5e8031e848781f34a5c22b9dc3647f94d309352e WatchSource:0}: Error finding container 3a34c96c843c4eab818c7f0d5e8031e848781f34a5c22b9dc3647f94d309352e: Status 404 returned error can't find the container with id 3a34c96c843c4eab818c7f0d5e8031e848781f34a5c22b9dc3647f94d309352e Oct 29 22:12:52.377201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.377155 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:12:52.377310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.377261 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:12:52.377351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.377308 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:12:52.420693 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.418977 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:52.519227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.519192 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" event={"ID":"938f28fc-c391-46db-aa26-cb709c6eb9ec","Type":"ContainerStarted","Data":"77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87"} Oct 29 22:12:52.519432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.519245 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" event={"ID":"938f28fc-c391-46db-aa26-cb709c6eb9ec","Type":"ContainerStarted","Data":"3a34c96c843c4eab818c7f0d5e8031e848781f34a5c22b9dc3647f94d309352e"} Oct 29 22:12:52.610012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.597298 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" podStartSLOduration=1.597275121 podStartE2EDuration="1.597275121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:52.585367717 +0000 UTC m=+1838.059956160" watchObservedRunningTime="2026-10-29 22:12:52.597275121 +0000 UTC m=+1838.071863546" Oct 29 22:12:52.728238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.728079 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:52.922172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:52.922130 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:53.009306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.006346 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:12:53.030938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.029809 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:12:53.030938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.029878 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:12:53.530366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.530326 4952 generic.go:356] "Generic (PLEG): container finished" podID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerID="89c48aa2572127a93362586fe8b79e2589467ae0dd691ceecae21b7997b6cae6" exitCode=0 Oct 29 22:12:53.530571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.530406 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" event={"ID":"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1","Type":"ContainerDied","Data":"89c48aa2572127a93362586fe8b79e2589467ae0dd691ceecae21b7997b6cae6"} Oct 29 22:12:53.531030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.531008 4952 scope.go:122] "RemoveContainer" containerID="89c48aa2572127a93362586fe8b79e2589467ae0dd691ceecae21b7997b6cae6" Oct 29 22:12:53.532942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.532876 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:12:53.533033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.533002 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:12:53.533082 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.533050 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:12:53.565438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.565411 4952 patch_prober.go:29] interesting pod/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:12:53.565583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:53.565463 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:12:54.001844 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:54.001813 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:12:54.002324 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:54.002298 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:12:54.282430 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:54.282335 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:54.479238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:54.479013 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:54.879651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:54.879616 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" event={"ID":"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1","Type":"ContainerStarted","Data":"6584e0e568a980d2bd5a42a2a970b8606d6ac856029c460e222598fdd6e4f45b"} Oct 29 22:12:55.002074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.001089 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:12:55.002074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.001755 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:12:55.002074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.001796 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:12:55.002892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.002774 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:12:55.002892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.002833 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:12:55.376787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.368841 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:55.612190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.612086 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:12:55.744983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.744407 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-envoy\") pod \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " Oct 29 22:12:55.744983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.744628 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-data\") pod \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " Oct 29 22:12:55.744983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.744703 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-podinfo\") pod \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " Oct 29 22:12:55.744983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.744748 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-token\" (UniqueName: \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-token\") pod \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " Oct 29 22:12:55.744983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.744789 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-credential-socket\") pod \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " Oct 29 22:12:55.744983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.744821 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-certs\") pod \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " Oct 29 22:12:55.744983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.744858 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istiod-ca-cert\") pod \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " Oct 29 22:12:55.745511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.744998 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-socket\") pod \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " Oct 29 22:12:55.745511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.745028 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-kube-api-access-hfb7l\" (UniqueName: \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-kube-api-access-hfb7l\") pod \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\" (UID: \"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35\") " Oct 29 22:12:55.795679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.795300 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-certs" pod "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" (UID: "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.795679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.795640 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-data" pod "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" (UID: "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.796833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.796809 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-socket" pod "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" (UID: "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.797476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.797453 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-credential-socket" pod "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" (UID: "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.801002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.800969 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istiod-ca-cert" pod "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" (UID: "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:12:55.802380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.802338 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-podinfo" pod "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" (UID: "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:12:55.803022 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.802992 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-envoy" pod "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" (UID: "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:12:55.805054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.804156 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-kube-api-access-hfb7l" pod "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" (UID: "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35"). InnerVolumeSpecName "kube-api-access-hfb7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:55.806318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.806290 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-token" pod "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" (UID: "ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:12:55.869644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.864822 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.869644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.864862 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfb7l\" (UniqueName: \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-kube-api-access-hfb7l\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.869644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.864877 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.869644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.864890 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.869644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.864921 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.869644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.864933 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.869644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.865009 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.869644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.865028 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.869644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.865042 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:12:55.897135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.897042 4952 generic.go:356] "Generic (PLEG): container finished" podID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerID="ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d" exitCode=0 Oct 29 22:12:55.898376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.898255 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" Oct 29 22:12:55.898630 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.898591 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" event={"ID":"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35","Type":"ContainerDied","Data":"ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d"} Oct 29 22:12:55.898786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.898771 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw" event={"ID":"ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35","Type":"ContainerDied","Data":"acecac2802e5a634bd98d5997d1bf685d07c47a3e7eedc5a9a8519393f72e2cf"} Oct 29 22:12:55.899037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.899022 4952 scope.go:122] "RemoveContainer" containerID="ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d" Oct 29 22:12:55.917561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.917534 4952 scope.go:122] "RemoveContainer" containerID="ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d" Oct 29 22:12:55.917985 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:12:55.917962 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d\": container with ID starting with ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d not found: ID does not exist" containerID="ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d" Oct 29 22:12:55.918054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.917993 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d"} err="failed to get container status \"ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d\": rpc error: code = NotFound desc = could not find container \"ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d\": container with ID starting with ba5e20019c5f8b44b635a5c1d4cc6fee879b284b641f62e4f1d593e4e63eee8d not found: ID does not exist" Oct 29 22:12:55.997375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:55.997285 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw"] Oct 29 22:12:56.001077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:56.001040 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:12:56.001319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:56.001096 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:12:56.001965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:56.001935 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-796d71-openshift-default-788f569497-5b2sw"] Oct 29 22:12:56.002137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:56.002116 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:12:56.002498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:56.002473 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:12:56.699709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:56.699142 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" path="/var/lib/kubelet/pods/ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35/volumes" Oct 29 22:12:57.004249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:57.004167 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:12:57.004712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:57.004686 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:12:57.004797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:57.004167 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:12:57.004862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:57.004851 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:12:57.735772 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:57.735732 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:58.001392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:58.001296 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:12:58.001562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:58.001415 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:12:58.001562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:58.001299 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:12:58.001562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:58.001496 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:12:58.018726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:58.018689 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:58.757737 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:58.757587 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:59.018541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:59.016502 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:12:59.018541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:59.016559 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 29 22:12:59.018541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:59.017035 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:12:59.018541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:59.017080 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:12:59.148504 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:59.148465 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:12:59.391931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:59.391482 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:12:59.997265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:12:59.997226 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:00.000921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:00.000872 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 29 22:13:00.001142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:00.001114 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 29 22:13:00.082187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:00.081376 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:13:00.083188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:00.083149 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:13:00.086706 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:00.086678 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:13:00.170857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:00.170810 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:01.017941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:01.017856 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:13:01.020396 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:01.020370 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:13:02.153211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:02.152572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:03.200120 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:03.200076 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:03.372552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:03.372516 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:03.785609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:03.785546 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:04.125285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:04.124399 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:04.536697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:04.536613 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:04.721045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:04.713852 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:05.119436 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:05.119399 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:05.297524 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:05.297484 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:06.226815 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:06.226777 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:06.961980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:06.961944 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:07.142045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:07.142008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:07.980445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:07.980405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:08.158870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:08.158826 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:08.347921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:08.342893 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:09.112942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:09.112868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:09.282554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:09.282510 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:09.717512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:09.717475 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:09.893464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:09.893410 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:10.576350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:10.576320 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:10.788110 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:10.788067 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:10.990502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:10.990417 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:11.227488 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:11.227451 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:12.807407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:12.807365 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:14.692139 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:14.692104 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:15.533055 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:13:15.533016 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:13:16.347922 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:16.347672 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:16.525600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:16.525562 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:17.863930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:17.859847 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:18.028789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:18.028754 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:18.209622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:18.209523 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:18.376924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:18.376628 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:19.869580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:19.869551 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:20.609964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:20.609868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:20.797302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:20.797255 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:20.988305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:20.988147 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:21.283008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:21.279443 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:23.121945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:23.109040 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:25.344934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:25.338726 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:27.047000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:27.046954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:27.936956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:27.932800 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:28.098423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:28.098388 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:28.764669 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:28.764636 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:28.932261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:28.932226 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:29.432363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:29.432328 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk"] Oct 29 22:13:29.606270 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:29.606238 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:29.793484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:29.793393 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:30.677822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:30.677752 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:30.869055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:30.869023 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:31.057998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:31.057784 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:32.594028 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:32.593991 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:33.104896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:33.104863 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:34.436033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:34.435960 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" podUID="ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1" containerName="payload-pre-processing" containerID="cri-o://46344336d75f4c95b09568a5021d933ddf6a7faad0ab564c9662bab6892fdf6d" gracePeriod=25 Oct 29 22:13:34.709777 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:34.709662 4952 generic.go:356] "Generic (PLEG): container finished" podID="ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1" containerID="46344336d75f4c95b09568a5021d933ddf6a7faad0ab564c9662bab6892fdf6d" exitCode=0 Oct 29 22:13:34.709777 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:34.709740 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" event={"ID":"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1","Type":"ContainerDied","Data":"46344336d75f4c95b09568a5021d933ddf6a7faad0ab564c9662bab6892fdf6d"} Oct 29 22:13:34.885395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:34.885366 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:13:34.971117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:34.971023 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-kube-api-access-mqq7r\" (UniqueName: \"kubernetes.io/projected/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-kube-api-access-mqq7r\") pod \"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1\" (UID: \"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1\") " Oct 29 22:13:34.971117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:34.971081 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-plugins-config-volume\") pod \"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1\" (UID: \"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1\") " Oct 29 22:13:34.987963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:34.982050 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-kube-api-access-mqq7r" pod "ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1" (UID: "ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1"). InnerVolumeSpecName "kube-api-access-mqq7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:34.987963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:34.983117 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-plugins-config-volume" pod "ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1" (UID: "ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:35.074145 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:35.074107 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqq7r\" (UniqueName: \"kubernetes.io/projected/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-kube-api-access-mqq7r\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:13:35.074145 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:35.074145 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:13:35.209924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:35.208971 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:35.726950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:35.723605 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" event={"ID":"ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1","Type":"ContainerDied","Data":"376765cc3fa675580c813f00f96c2e40c5c33d26304989765846c29844038a83"} Oct 29 22:13:35.726950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:35.723685 4952 scope.go:122] "RemoveContainer" containerID="46344336d75f4c95b09568a5021d933ddf6a7faad0ab564c9662bab6892fdf6d" Oct 29 22:13:35.726950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:35.723865 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk" Oct 29 22:13:35.784936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:35.784884 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk"] Oct 29 22:13:35.796230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:35.796193 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4c5549-59bd75847f-gnrqk"] Oct 29 22:13:36.050693 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.050502 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:36.650145 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.650068 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1" path="/var/lib/kubelet/pods/ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1/volumes" Oct 29 22:13:36.755094 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.755057 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn"] Oct 29 22:13:36.756042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.756012 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1" containerName="payload-pre-processing" Oct 29 22:13:36.756042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.756039 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1" containerName="payload-pre-processing" Oct 29 22:13:36.756264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.756067 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" Oct 29 22:13:36.756264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.756075 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" Oct 29 22:13:36.756373 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.756266 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd5575c-0897-4cd9-8b95-dfb6cf7e7cc1" containerName="payload-pre-processing" Oct 29 22:13:36.756373 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.756283 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcc1f35-7d6c-45f9-8bdc-87ebfeeacd35" containerName="istio-proxy" Oct 29 22:13:36.761915 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.761881 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:36.794260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.794231 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn"] Oct 29 22:13:36.914048 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.913955 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:36.914214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.914090 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:36.914214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.914179 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:36.914331 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.914299 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c932aaf0-9272-4567-aaaa-90a392720191-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:36.914394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.914342 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:36.914489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.914471 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpphr\" (UniqueName: \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-kube-api-access-vpphr\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:36.914560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.914544 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:36.914676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.914659 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:36.914728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:36.914695 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c932aaf0-9272-4567-aaaa-90a392720191-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.015824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.015786 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.015824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.015825 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c932aaf0-9272-4567-aaaa-90a392720191-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016087 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.015877 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016087 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.015934 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016087 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.015993 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016087 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.016057 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c932aaf0-9272-4567-aaaa-90a392720191-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.016088 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.016162 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpphr\" (UniqueName: \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-kube-api-access-vpphr\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.016230 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.016293 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016528 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.016477 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.016602 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016711 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.016648 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.016782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.016769 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c932aaf0-9272-4567-aaaa-90a392720191-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.018698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.018673 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.018989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.018972 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c932aaf0-9272-4567-aaaa-90a392720191-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.040815 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.040783 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpphr\" (UniqueName: \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-kube-api-access-vpphr\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.041037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.041018 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.093156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.093126 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:37.473595 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:13:37.473564 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc932aaf0_9272_4567_aaaa_90a392720191.slice/crio-2789279d4e6c2c3a78a0606d0f4d49b0125fe967758ffec18a39bcd6a2b5d21f WatchSource:0}: Error finding container 2789279d4e6c2c3a78a0606d0f4d49b0125fe967758ffec18a39bcd6a2b5d21f: Status 404 returned error can't find the container with id 2789279d4e6c2c3a78a0606d0f4d49b0125fe967758ffec18a39bcd6a2b5d21f Oct 29 22:13:37.474610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.474579 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn"] Oct 29 22:13:37.476821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.476778 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:13:37.477007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.476861 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:13:37.477007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.476900 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:13:37.543344 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.543279 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:37.748738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.748642 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" event={"ID":"c932aaf0-9272-4567-aaaa-90a392720191","Type":"ContainerStarted","Data":"bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71"} Oct 29 22:13:37.748738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.748706 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" event={"ID":"c932aaf0-9272-4567-aaaa-90a392720191","Type":"ContainerStarted","Data":"2789279d4e6c2c3a78a0606d0f4d49b0125fe967758ffec18a39bcd6a2b5d21f"} Oct 29 22:13:37.829240 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:37.820814 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" podStartSLOduration=1.820795104 podStartE2EDuration="1.820795104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:37.82012634 +0000 UTC m=+1883.294714754" watchObservedRunningTime="2026-10-29 22:13:37.820795104 +0000 UTC m=+1883.295383522" Oct 29 22:13:38.094308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:38.094267 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:38.097948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:38.097358 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:13:38.097948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:38.097420 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:13:39.094356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.094324 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:13:39.094740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.094380 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:13:39.164931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.164636 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2"] Oct 29 22:13:39.170411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.170389 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.193960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.192262 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2"] Oct 29 22:13:39.196105 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.196074 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-971004-serving-cert\"" Oct 29 22:13:39.310333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.310299 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:39.369278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.369184 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-maas-api-tls\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.379933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.376722 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxp2t\" (UniqueName: \"kubernetes.io/projected/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-kube-api-access-wxp2t\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.386291 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.386252 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-metrics-certs\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.386605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.386585 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-openshift-service-ca\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.486205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.485786 4952 patch_prober.go:29] interesting pod/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:13:39.486205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.485838 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:13:39.488897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.488860 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxp2t\" (UniqueName: \"kubernetes.io/projected/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-kube-api-access-wxp2t\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.489076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.489058 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-metrics-certs\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.489132 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.489116 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-openshift-service-ca\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.489277 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.489261 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-maas-api-tls\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.490464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.490440 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-openshift-service-ca\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.493195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.493150 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-metrics-certs\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.493704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.493676 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-maas-api-tls\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.494345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.494314 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:39.515351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.515319 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxp2t\" (UniqueName: \"kubernetes.io/projected/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-kube-api-access-wxp2t\") pod \"maas-api-e2e-autores-971004-b44bd4f7-nk8x2\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:39.676405 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.676303 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:39.791438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:39.791405 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:40.117068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:40.110143 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:13:40.117068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:40.110205 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:13:40.186482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:40.186447 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2"] Oct 29 22:13:40.803721 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:40.803674 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" event={"ID":"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca","Type":"ContainerStarted","Data":"778ef0e01c6ecf23d07e59fb9e213434fd9852b8531f9edfd9e552500036eeaf"} Oct 29 22:13:41.095385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:41.095331 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:13:41.095604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:41.095397 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:13:41.366985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:41.365997 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:41.829231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:41.829111 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" event={"ID":"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca","Type":"ContainerStarted","Data":"617866489a9ec9463a1170119181595771f4b1851be5c1778a5001066b1ae289"} Oct 29 22:13:41.830840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:41.830794 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:41.886108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:41.886039 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" podStartSLOduration=2.359153497 podStartE2EDuration="2.886018882s" totalImagesPullingTime="526.865385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:41.88235174 +0000 UTC m=+1887.356940159" watchObservedRunningTime="2026-10-29 22:13:41.886018882 +0000 UTC m=+1887.360607301" Oct 29 22:13:42.096023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:42.095987 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:13:42.096312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:42.096282 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:13:42.860930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:42.860331 4952 generic.go:356] "Generic (PLEG): container finished" podID="edadf27c-b54d-403b-8618-e240e0b9f728" containerID="e7552e8bcc313659fed883b6f3e682bac8915fa9ab907819bf5ece9f9a1c9ec1" exitCode=0 Oct 29 22:13:42.861453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:42.861095 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" event={"ID":"edadf27c-b54d-403b-8618-e240e0b9f728","Type":"ContainerDied","Data":"e7552e8bcc313659fed883b6f3e682bac8915fa9ab907819bf5ece9f9a1c9ec1"} Oct 29 22:13:42.863935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:42.861572 4952 scope.go:122] "RemoveContainer" containerID="e7552e8bcc313659fed883b6f3e682bac8915fa9ab907819bf5ece9f9a1c9ec1" Oct 29 22:13:42.863935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:42.863718 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:13:42.863935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:42.863808 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:13:42.863935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:42.863851 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:13:43.093542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:43.093504 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 29 22:13:43.093823 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:43.093790 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 29 22:13:43.295005 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:43.290596 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:43.643828 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:43.643657 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:43.812153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:43.812114 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:43.873048 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:43.872940 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" event={"ID":"edadf27c-b54d-403b-8618-e240e0b9f728","Type":"ContainerStarted","Data":"a1d963daa6e3194093272c8308e4695b76d136969afb1ee32568f27d22a4caa4"} Oct 29 22:13:43.898026 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:43.897950 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:13:44.100004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:44.099692 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:44.101267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:44.101236 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:44.103814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:44.103787 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:13:44.897848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:44.897813 4952 patch_prober.go:29] interesting pod/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:13:44.898316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:44.897871 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 29 22:13:45.132576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:45.132538 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:45.554786 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:13:45.554729 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:13:45.898473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:45.897790 4952 patch_prober.go:29] interesting pod/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:13:45.898473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:45.897847 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 29 22:13:46.898223 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:46.898187 4952 patch_prober.go:29] interesting pod/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:13:46.898450 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:46.898263 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 29 22:13:46.969617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:46.969578 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:47.898101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:47.898064 4952 patch_prober.go:29] interesting pod/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:13:47.898280 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:47.898140 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 29 22:13:48.874868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:48.874833 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:48.899095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:48.899060 4952 patch_prober.go:29] interesting pod/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:13:48.899495 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:48.899378 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 29 22:13:48.901537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:48.901515 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:13:49.902841 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:49.902807 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:13:49.903359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:49.903336 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:13:49.904133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:49.904108 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:13:50.247147 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:50.247059 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696"] Oct 29 22:13:50.247601 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:50.247554 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" podUID="01abd238-8ef8-4e3e-9503-49304f28490b" containerName="maas-api" containerID="cri-o://7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1" gracePeriod=30 Oct 29 22:13:50.368583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:50.368550 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:51.066347 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.065858 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:13:51.068771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.068159 4952 generic.go:356] "Generic (PLEG): container finished" podID="01abd238-8ef8-4e3e-9503-49304f28490b" containerID="7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1" exitCode=0 Oct 29 22:13:51.069499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.069473 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" event={"ID":"01abd238-8ef8-4e3e-9503-49304f28490b","Type":"ContainerDied","Data":"7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1"} Oct 29 22:13:51.069681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.069660 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" event={"ID":"01abd238-8ef8-4e3e-9503-49304f28490b","Type":"ContainerDied","Data":"7430bdfc54af0f7800bcde9ea96a0740abec4885751bd069cbf8b309041f5e23"} Oct 29 22:13:51.069802 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.069787 4952 scope.go:122] "RemoveContainer" containerID="7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1" Oct 29 22:13:51.096244 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.096217 4952 scope.go:122] "RemoveContainer" containerID="7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1" Oct 29 22:13:51.099131 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:13:51.099056 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1\": container with ID starting with 7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1 not found: ID does not exist" containerID="7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1" Oct 29 22:13:51.099131 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.099101 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1"} err="failed to get container status \"7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1\": rpc error: code = NotFound desc = could not find container \"7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1\": container with ID starting with 7eb8ad9e90c25fa6d0f23cc4e80cab15a8f4ee5ea506ca70549d68be6ea75fb1 not found: ID does not exist" Oct 29 22:13:51.137042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.137005 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01abd238-8ef8-4e3e-9503-49304f28490b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01abd238-8ef8-4e3e-9503-49304f28490b-openshift-service-ca\") pod \"01abd238-8ef8-4e3e-9503-49304f28490b\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " Oct 29 22:13:51.139157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.139127 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-metrics-certs\") pod \"01abd238-8ef8-4e3e-9503-49304f28490b\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " Oct 29 22:13:51.139865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.139843 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01abd238-8ef8-4e3e-9503-49304f28490b-kube-api-access-bb88r\" (UniqueName: \"kubernetes.io/projected/01abd238-8ef8-4e3e-9503-49304f28490b-kube-api-access-bb88r\") pod \"01abd238-8ef8-4e3e-9503-49304f28490b\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " Oct 29 22:13:51.140635 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.140617 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-maas-api-tls\") pod \"01abd238-8ef8-4e3e-9503-49304f28490b\" (UID: \"01abd238-8ef8-4e3e-9503-49304f28490b\") " Oct 29 22:13:51.143132 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.137612 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01abd238-8ef8-4e3e-9503-49304f28490b-openshift-service-ca" pod "01abd238-8ef8-4e3e-9503-49304f28490b" (UID: "01abd238-8ef8-4e3e-9503-49304f28490b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:13:51.191036 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.190977 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-maas-api-tls" pod "01abd238-8ef8-4e3e-9503-49304f28490b" (UID: "01abd238-8ef8-4e3e-9503-49304f28490b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:13:51.194954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.194175 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-metrics-certs" pod "01abd238-8ef8-4e3e-9503-49304f28490b" (UID: "01abd238-8ef8-4e3e-9503-49304f28490b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:13:51.194954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.194712 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01abd238-8ef8-4e3e-9503-49304f28490b-kube-api-access-bb88r" pod "01abd238-8ef8-4e3e-9503-49304f28490b" (UID: "01abd238-8ef8-4e3e-9503-49304f28490b"). InnerVolumeSpecName "kube-api-access-bb88r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:13:51.243741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.243613 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb88r\" (UniqueName: \"kubernetes.io/projected/01abd238-8ef8-4e3e-9503-49304f28490b-kube-api-access-bb88r\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:13:51.243741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.243652 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:13:51.243741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.243672 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01abd238-8ef8-4e3e-9503-49304f28490b-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:13:51.243741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.243686 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01abd238-8ef8-4e3e-9503-49304f28490b-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:13:51.573238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.573146 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:51.763930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.762894 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:51.788962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.788921 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn"] Oct 29 22:13:51.789740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.789707 4952 cpu_manager.go:414] "containerMap: removing container" podUID="01abd238-8ef8-4e3e-9503-49304f28490b" containerName="maas-api" Oct 29 22:13:51.789740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.789730 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01abd238-8ef8-4e3e-9503-49304f28490b" containerName="maas-api" Oct 29 22:13:51.789971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.789840 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="01abd238-8ef8-4e3e-9503-49304f28490b" containerName="maas-api" Oct 29 22:13:51.795293 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.795269 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.838292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.838261 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn"] Oct 29 22:13:51.864435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.864396 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.864629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.864551 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.864629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.864622 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.864801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.864693 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg5wv\" (UniqueName: \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-kube-api-access-qg5wv\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.864801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.864761 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.864989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.864942 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.865087 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.864985 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.865087 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.865024 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.865087 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.865067 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.966655 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.966620 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.966833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.966697 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.966833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.966780 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5wv\" (UniqueName: \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-kube-api-access-qg5wv\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.966833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.966810 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.967128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.967022 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.967128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.967069 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.967128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.967111 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.967344 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.967160 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.967344 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.967209 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.967344 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.967253 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-data\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.967966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.967440 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istiod-ca-cert\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.968127 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.968097 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-certs\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.968388 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.968349 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-credential-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.968388 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.968347 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:51.968754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.968729 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-socket\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.970321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.970295 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-podinfo\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.970497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.970478 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-envoy\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.992092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.992061 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-token\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:51.992630 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:51.992607 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg5wv\" (UniqueName: \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-kube-api-access-qg5wv\") pod \"e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:52.076338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:52.076309 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696" Oct 29 22:13:52.113023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:52.112946 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:52.137300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:52.137268 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696"] Oct 29 22:13:52.144398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:52.144372 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4c5549-5d8b5b8767-9h696"] Oct 29 22:13:52.338016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:52.337980 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:52.453072 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:52.453045 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn"] Oct 29 22:13:52.455003 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:13:52.454968 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6add6ee_6c29_4d7e_94f1_ddb3698c1244.slice/crio-3e473fe6f6f2f10cae8cadc62c4bca69ca44a38e4bcce0564d68c3ef42b9d079 WatchSource:0}: Error finding container 3e473fe6f6f2f10cae8cadc62c4bca69ca44a38e4bcce0564d68c3ef42b9d079: Status 404 returned error can't find the container with id 3e473fe6f6f2f10cae8cadc62c4bca69ca44a38e4bcce0564d68c3ef42b9d079 Oct 29 22:13:52.458343 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:52.458311 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:13:52.458447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:52.458374 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:13:52.458447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:52.458404 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:13:52.650076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:52.649788 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01abd238-8ef8-4e3e-9503-49304f28490b" path="/var/lib/kubelet/pods/01abd238-8ef8-4e3e-9503-49304f28490b/volumes" Oct 29 22:13:53.088268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:53.088224 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" event={"ID":"e6add6ee-6c29-4d7e-94f1-ddb3698c1244","Type":"ContainerStarted","Data":"bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe"} Oct 29 22:13:53.088724 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:53.088336 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" event={"ID":"e6add6ee-6c29-4d7e-94f1-ddb3698c1244","Type":"ContainerStarted","Data":"3e473fe6f6f2f10cae8cadc62c4bca69ca44a38e4bcce0564d68c3ef42b9d079"} Oct 29 22:13:53.127936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:53.115459 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:53.127936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:53.115679 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:13:53.127936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:53.115759 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 22:13:53.152424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:53.152355 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" podStartSLOduration=2.15233143 podStartE2EDuration="2.15233143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:53.134372012 +0000 UTC m=+1898.608960430" watchObservedRunningTime="2026-10-29 22:13:53.15233143 +0000 UTC m=+1898.626919849" Oct 29 22:13:53.945928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:53.940411 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:54.114374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:54.114340 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:13:54.114774 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:54.114413 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 22:13:54.560926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:54.544832 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:55.113379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:55.113337 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:13:55.113636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:55.113395 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 22:13:55.447990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:55.445606 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:55.953933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:55.950102 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:56.126643 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:56.126603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:56.148103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:56.147964 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:13:56.149004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:56.148970 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 22:13:56.401935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:56.399455 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:56.704039 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:56.691739 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:56.880929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:56.880726 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:57.113410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:57.113365 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:13:57.113588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:57.113432 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 22:13:58.111978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:58.111941 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:58.116294 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:58.116261 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:13:58.116562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:58.116530 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 22:13:59.264474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:59.264440 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:59.265787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:59.265762 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:59.269991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:59.267109 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:13:59.453184 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:59.453148 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:59.615893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:59.615858 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:13:59.726381 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:59.726348 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf"] Oct 29 22:13:59.726838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:59.726807 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" containerID="cri-o://a1d963daa6e3194093272c8308e4695b76d136969afb1ee32568f27d22a4caa4" gracePeriod=30 Oct 29 22:13:59.858927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:13:59.851444 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:00.249949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.249876 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:00.561675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.561588 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7758-11d2bf"] Oct 29 22:14:00.567883 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.567858 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-11d2bf" Oct 29 22:14:00.585310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.585282 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r84ws\"" Oct 29 22:14:00.587492 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.587412 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7758-11d2bf"] Oct 29 22:14:00.640711 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.640672 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:00.665704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.665672 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhwj\" (UniqueName: \"kubernetes.io/projected/fb5a1687-ae02-4465-b16b-d70ef37dd322-kube-api-access-xvhwj\") pod \"test-curl-7758-11d2bf\" (UID: \"fb5a1687-ae02-4465-b16b-d70ef37dd322\") " pod="opendatahub/test-curl-7758-11d2bf" Oct 29 22:14:00.767453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.767410 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhwj\" (UniqueName: \"kubernetes.io/projected/fb5a1687-ae02-4465-b16b-d70ef37dd322-kube-api-access-xvhwj\") pod \"test-curl-7758-11d2bf\" (UID: \"fb5a1687-ae02-4465-b16b-d70ef37dd322\") " pod="opendatahub/test-curl-7758-11d2bf" Oct 29 22:14:00.793529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.793480 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhwj\" (UniqueName: \"kubernetes.io/projected/fb5a1687-ae02-4465-b16b-d70ef37dd322-kube-api-access-xvhwj\") pod \"test-curl-7758-11d2bf\" (UID: \"fb5a1687-ae02-4465-b16b-d70ef37dd322\") " pod="opendatahub/test-curl-7758-11d2bf" Oct 29 22:14:00.826827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.826795 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:00.884160 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:00.884132 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-11d2bf" Oct 29 22:14:01.209245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:01.209157 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7758-11d2bf"] Oct 29 22:14:01.215545 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:14:01.215516 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb5a1687_ae02_4465_b16b_d70ef37dd322.slice/crio-870091ab18273560db1c0adec41cf4b8d365ae094df11f6b7dcdd67d19407ae7 WatchSource:0}: Error finding container 870091ab18273560db1c0adec41cf4b8d365ae094df11f6b7dcdd67d19407ae7: Status 404 returned error can't find the container with id 870091ab18273560db1c0adec41cf4b8d365ae094df11f6b7dcdd67d19407ae7 Oct 29 22:14:01.671306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:01.666982 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:14:02.225474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:02.225437 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7758-11d2bf" event={"ID":"fb5a1687-ae02-4465-b16b-d70ef37dd322","Type":"ContainerStarted","Data":"32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179"} Oct 29 22:14:02.225674 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:02.225487 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7758-11d2bf" event={"ID":"fb5a1687-ae02-4465-b16b-d70ef37dd322","Type":"ContainerStarted","Data":"870091ab18273560db1c0adec41cf4b8d365ae094df11f6b7dcdd67d19407ae7"} Oct 29 22:14:02.266392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:02.266248 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7758-11d2bf" podStartSLOduration=2.266229772 podStartE2EDuration="2.266229772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:02.265364427 +0000 UTC m=+1907.739952885" watchObservedRunningTime="2026-10-29 22:14:02.266229772 +0000 UTC m=+1907.740818191" Oct 29 22:14:02.379820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:02.379781 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:02.554374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:02.554300 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:02.791940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:02.777985 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:04.227977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:04.227941 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7758-11d2bf"] Oct 29 22:14:04.236853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:04.236817 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7758-11d2bf"] Oct 29 22:14:04.250736 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:04.250137 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7758-11d2bf" podUID="fb5a1687-ae02-4465-b16b-d70ef37dd322" containerName="test-curl-7758-11d2bf" containerID="cri-o://32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179" gracePeriod=2 Oct 29 22:14:04.283452 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:14:04.283392 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7758-11d2bf\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="fb5a1687-ae02-4465-b16b-d70ef37dd322" pod="opendatahub/test-curl-7758-11d2bf" Oct 29 22:14:04.831928 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:14:04.828295 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7758-11d2bf\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="fb5a1687-ae02-4465-b16b-d70ef37dd322" pod="opendatahub/test-curl-7758-11d2bf" Oct 29 22:14:04.904708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:04.904671 4952 patch_prober.go:29] interesting pod/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:14:04.905011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:04.904979 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 29 22:14:05.108928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.107924 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:05.291862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.290008 4952 generic.go:356] "Generic (PLEG): container finished" podID="edadf27c-b54d-403b-8618-e240e0b9f728" containerID="a1d963daa6e3194093272c8308e4695b76d136969afb1ee32568f27d22a4caa4" exitCode=0 Oct 29 22:14:05.291862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.290404 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" event={"ID":"edadf27c-b54d-403b-8618-e240e0b9f728","Type":"ContainerDied","Data":"a1d963daa6e3194093272c8308e4695b76d136969afb1ee32568f27d22a4caa4"} Oct 29 22:14:05.291862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.290674 4952 scope.go:122] "RemoveContainer" containerID="e7552e8bcc313659fed883b6f3e682bac8915fa9ab907819bf5ece9f9a1c9ec1" Oct 29 22:14:05.453943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.447043 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:05.471666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.471530 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:14:05.566093 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.565326 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-envoy\") pod \"edadf27c-b54d-403b-8618-e240e0b9f728\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " Oct 29 22:14:05.566093 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.565858 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edadf27c-b54d-403b-8618-e240e0b9f728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edadf27c-b54d-403b-8618-e240e0b9f728-istio-podinfo\") pod \"edadf27c-b54d-403b-8618-e240e0b9f728\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " Oct 29 22:14:05.566093 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.566005 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-credential-socket\") pod \"edadf27c-b54d-403b-8618-e240e0b9f728\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " Oct 29 22:14:05.566369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.566110 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-istio-token\" (UniqueName: \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-istio-token\") pod \"edadf27c-b54d-403b-8618-e240e0b9f728\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " Oct 29 22:14:05.566369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.566165 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-data\") pod \"edadf27c-b54d-403b-8618-e240e0b9f728\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " Oct 29 22:14:05.566369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.566198 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edadf27c-b54d-403b-8618-e240e0b9f728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edadf27c-b54d-403b-8618-e240e0b9f728-istiod-ca-cert\") pod \"edadf27c-b54d-403b-8618-e240e0b9f728\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " Oct 29 22:14:05.566369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.566258 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-certs\") pod \"edadf27c-b54d-403b-8618-e240e0b9f728\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " Oct 29 22:14:05.566369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.566316 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-kube-api-access-frp2z\" (UniqueName: \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-kube-api-access-frp2z\") pod \"edadf27c-b54d-403b-8618-e240e0b9f728\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " Oct 29 22:14:05.566661 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.566443 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-socket\") pod \"edadf27c-b54d-403b-8618-e240e0b9f728\" (UID: \"edadf27c-b54d-403b-8618-e240e0b9f728\") " Oct 29 22:14:05.567411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.567082 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-data" pod "edadf27c-b54d-403b-8618-e240e0b9f728" (UID: "edadf27c-b54d-403b-8618-e240e0b9f728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:14:05.568530 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.568501 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:14:05.568941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.568922 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-socket" pod "edadf27c-b54d-403b-8618-e240e0b9f728" (UID: "edadf27c-b54d-403b-8618-e240e0b9f728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:14:05.569401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.569383 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edadf27c-b54d-403b-8618-e240e0b9f728-istiod-ca-cert" pod "edadf27c-b54d-403b-8618-e240e0b9f728" (UID: "edadf27c-b54d-403b-8618-e240e0b9f728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:14:05.569782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.569763 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-certs" pod "edadf27c-b54d-403b-8618-e240e0b9f728" (UID: "edadf27c-b54d-403b-8618-e240e0b9f728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:14:05.586980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.583305 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-credential-socket" pod "edadf27c-b54d-403b-8618-e240e0b9f728" (UID: "edadf27c-b54d-403b-8618-e240e0b9f728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:14:05.603523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.603483 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-kube-api-access-frp2z" pod "edadf27c-b54d-403b-8618-e240e0b9f728" (UID: "edadf27c-b54d-403b-8618-e240e0b9f728"). InnerVolumeSpecName "kube-api-access-frp2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:14:05.605535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.605479 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edadf27c-b54d-403b-8618-e240e0b9f728-istio-podinfo" pod "edadf27c-b54d-403b-8618-e240e0b9f728" (UID: "edadf27c-b54d-403b-8618-e240e0b9f728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:14:05.605794 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.605761 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-envoy" pod "edadf27c-b54d-403b-8618-e240e0b9f728" (UID: "edadf27c-b54d-403b-8618-e240e0b9f728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:14:05.631324 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.631261 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-istio-token" pod "edadf27c-b54d-403b-8618-e240e0b9f728" (UID: "edadf27c-b54d-403b-8618-e240e0b9f728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:14:05.682822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.682727 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:14:05.692129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.692098 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edadf27c-b54d-403b-8618-e240e0b9f728-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:14:05.692353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.692338 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:14:05.692452 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.692439 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:14:05.717719 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.716954 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edadf27c-b54d-403b-8618-e240e0b9f728-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:14:05.717719 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.716986 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:14:05.717719 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.717002 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frp2z\" (UniqueName: \"kubernetes.io/projected/edadf27c-b54d-403b-8618-e240e0b9f728-kube-api-access-frp2z\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:14:05.717719 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.717021 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edadf27c-b54d-403b-8618-e240e0b9f728-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:14:05.717719 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:05.717082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:06.321173 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:06.321121 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" event={"ID":"edadf27c-b54d-403b-8618-e240e0b9f728","Type":"ContainerDied","Data":"7a79fd5fcc86cfbfda6f3b219f5a7d792483a01d62e6f179f29ef8f1cb6c9927"} Oct 29 22:14:06.323067 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:06.323039 4952 scope.go:122] "RemoveContainer" containerID="a1d963daa6e3194093272c8308e4695b76d136969afb1ee32568f27d22a4caa4" Oct 29 22:14:06.323440 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:06.323422 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf" Oct 29 22:14:06.408855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:06.408818 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf"] Oct 29 22:14:06.421353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:06.421317 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4c5549-openshift-default-55c4ff96bf-qc2hf"] Oct 29 22:14:06.594990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:06.578203 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:06.674149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:06.672230 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" path="/var/lib/kubelet/pods/edadf27c-b54d-403b-8618-e240e0b9f728/volumes" Oct 29 22:14:07.180780 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:07.180745 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-11d2bf" Oct 29 22:14:07.280857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:07.280797 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5a1687-ae02-4465-b16b-d70ef37dd322-kube-api-access-xvhwj\" (UniqueName: \"kubernetes.io/projected/fb5a1687-ae02-4465-b16b-d70ef37dd322-kube-api-access-xvhwj\") pod \"fb5a1687-ae02-4465-b16b-d70ef37dd322\" (UID: \"fb5a1687-ae02-4465-b16b-d70ef37dd322\") " Oct 29 22:14:07.283899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:07.283861 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5a1687-ae02-4465-b16b-d70ef37dd322-kube-api-access-xvhwj" pod "fb5a1687-ae02-4465-b16b-d70ef37dd322" (UID: "fb5a1687-ae02-4465-b16b-d70ef37dd322"). InnerVolumeSpecName "kube-api-access-xvhwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:14:07.339926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:07.336150 4952 generic.go:356] "Generic (PLEG): container finished" podID="fb5a1687-ae02-4465-b16b-d70ef37dd322" containerID="32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179" exitCode=137 Oct 29 22:14:07.339926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:07.336398 4952 scope.go:122] "RemoveContainer" containerID="32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179" Oct 29 22:14:07.339926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:07.336508 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-11d2bf" Oct 29 22:14:07.383723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:07.382077 4952 scope.go:122] "RemoveContainer" containerID="32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179" Oct 29 22:14:07.383723 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:14:07.382309 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7758-11d2bf\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="fb5a1687-ae02-4465-b16b-d70ef37dd322" pod="opendatahub/test-curl-7758-11d2bf" Oct 29 22:14:07.383723 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:14:07.383249 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179\": container with ID starting with 32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179 not found: ID does not exist" containerID="32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179" Oct 29 22:14:07.383723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:07.383290 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179"} err="failed to get container status \"32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179\": rpc error: code = NotFound desc = could not find container \"32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179\": container with ID starting with 32bd0614ff2fbf1fe7460614ed742a483df2dfc186b2711827e245d9aa327179 not found: ID does not exist" Oct 29 22:14:07.383723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:07.383418 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvhwj\" (UniqueName: \"kubernetes.io/projected/fb5a1687-ae02-4465-b16b-d70ef37dd322-kube-api-access-xvhwj\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:14:07.607076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:07.607038 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:08.647898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:08.647862 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5a1687-ae02-4465-b16b-d70ef37dd322" path="/var/lib/kubelet/pods/fb5a1687-ae02-4465-b16b-d70ef37dd322/volumes" Oct 29 22:14:09.234491 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.234453 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:09.293130 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.293091 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n"] Oct 29 22:14:09.294172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.294147 4952 cpu_manager.go:414] "containerMap: removing container" podUID="fb5a1687-ae02-4465-b16b-d70ef37dd322" containerName="test-curl-7758-11d2bf" Oct 29 22:14:09.294283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.294174 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5a1687-ae02-4465-b16b-d70ef37dd322" containerName="test-curl-7758-11d2bf" Oct 29 22:14:09.294283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.294204 4952 cpu_manager.go:414] "containerMap: removing container" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" Oct 29 22:14:09.294283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.294213 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" Oct 29 22:14:09.294414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.294400 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" Oct 29 22:14:09.294447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.294423 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" Oct 29 22:14:09.294476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.294447 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5a1687-ae02-4465-b16b-d70ef37dd322" containerName="test-curl-7758-11d2bf" Oct 29 22:14:09.300108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.300084 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.327933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.326186 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n"] Oct 29 22:14:09.327933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.327073 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-99e420\"/\"openshift-service-ca.crt\"" Oct 29 22:14:09.327933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.327109 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-99e420\"/\"kube-root-ca.crt\"" Oct 29 22:14:09.333357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.333248 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-99e420\"/\"default-dockercfg-n2x4w\"" Oct 29 22:14:09.333357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.333329 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-99e420\"/\"auth-test-model-99e420-kserve-self-signed-certs\"" Oct 29 22:14:09.409250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.408658 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:09.423750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.423716 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-home\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.423955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.423792 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqshk\" (UniqueName: \"kubernetes.io/projected/cdaaa603-46cc-43ed-a102-9005c08df63f-kube-api-access-vqshk\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.424105 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.423957 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdaaa603-46cc-43ed-a102-9005c08df63f-tls-certs\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.424105 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.424004 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-tmp-dir\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.424105 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.424038 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-model-cache\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.424286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.424153 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-dshm\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.525986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.525856 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqshk\" (UniqueName: \"kubernetes.io/projected/cdaaa603-46cc-43ed-a102-9005c08df63f-kube-api-access-vqshk\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.525986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.525976 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdaaa603-46cc-43ed-a102-9005c08df63f-tls-certs\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.526229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.526007 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-tmp-dir\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.526229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.526027 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-model-cache\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.526229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.526102 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-dshm\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.526229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.526178 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-home\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.526552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.526526 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-home\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.526803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.526785 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-model-cache\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.526880 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.526800 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-tmp-dir\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.529448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.529421 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-dshm\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.530153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.530127 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdaaa603-46cc-43ed-a102-9005c08df63f-tls-certs\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.552207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.552173 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqshk\" (UniqueName: \"kubernetes.io/projected/cdaaa603-46cc-43ed-a102-9005c08df63f-kube-api-access-vqshk\") pod \"auth-test-model-99e420-kserve-f54c767c7-hmn6n\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:09.581218 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.581039 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:09.652076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:09.651782 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:10.103261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:10.103224 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:10.139680 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:10.139640 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n"] Oct 29 22:14:10.274119 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:10.274078 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:10.382253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:10.382214 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" event={"ID":"cdaaa603-46cc-43ed-a102-9005c08df63f","Type":"ContainerStarted","Data":"68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292"} Oct 29 22:14:10.382790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:10.382764 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" event={"ID":"cdaaa603-46cc-43ed-a102-9005c08df63f","Type":"ContainerStarted","Data":"08d1ec9275106a77aa2020241b0d2e7eceb6bae08ac09f1bb9e41a579ee995f6"} Oct 29 22:14:10.441849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:10.441729 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" podStartSLOduration=1.4417087 podStartE2EDuration="1.4417087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:10.428287657 +0000 UTC m=+1915.902876076" watchObservedRunningTime="2026-10-29 22:14:10.4417087 +0000 UTC m=+1915.916297119" Oct 29 22:14:10.444543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:10.443736 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:10.655923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:10.653670 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:10.908370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:10.908331 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:11.305199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.305127 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc"] Oct 29 22:14:11.305815 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.305798 4952 cpu_manager.go:414] "containerMap: removing container" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" Oct 29 22:14:11.305899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.305816 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edadf27c-b54d-403b-8618-e240e0b9f728" containerName="istio-proxy" Oct 29 22:14:11.309963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.309937 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.327796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.327757 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-8d569e-openshift-default-dockercfg-b69qr\"" Oct 29 22:14:11.351644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.351599 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc"] Oct 29 22:14:11.473739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.473703 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-certs\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.473969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.473779 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istiod-ca-cert\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.473969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.473924 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-envoy\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.474104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.473985 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-podinfo\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.474104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.474023 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-data\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.474224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.474150 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-credential-socket\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.474224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.474205 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-token\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.474339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.474317 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-socket\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.474398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.474356 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkwbq\" (UniqueName: \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-kube-api-access-tkwbq\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.578981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.565578 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:11.583937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.583634 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-certs\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.583937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.583730 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istiod-ca-cert\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.583937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.583833 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-envoy\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.583937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.583893 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-podinfo\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.584287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.583953 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-data\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.584287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.584013 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-credential-socket\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.584287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.584042 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-token\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.584287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.584083 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-socket\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.584287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.584127 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkwbq\" (UniqueName: \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-kube-api-access-tkwbq\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.587924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.584999 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-certs\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.587924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.585668 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istiod-ca-cert\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.587924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.586592 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-credential-socket\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.587924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.587575 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-socket\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.608924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.598221 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-data\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.628936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.628667 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-token\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.638929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.633401 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-podinfo\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.638929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.633704 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-envoy\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.666929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.664745 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkwbq\" (UniqueName: \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-kube-api-access-tkwbq\") pod \"e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:11.753015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.749661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:11.925231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.925131 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:11.962276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:11.962239 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:12.169011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:12.147724 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:12.537646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:12.537614 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc"] Oct 29 22:14:12.540543 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:14:12.540506 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345f5596_f75f_4aa1_adc4_d97f16ffa1a0.slice/crio-04ebe3583d11551286f3cbc9eaa0e1276c5a4af208d68728160b0dcce07350ba WatchSource:0}: Error finding container 04ebe3583d11551286f3cbc9eaa0e1276c5a4af208d68728160b0dcce07350ba: Status 404 returned error can't find the container with id 04ebe3583d11551286f3cbc9eaa0e1276c5a4af208d68728160b0dcce07350ba Oct 29 22:14:12.544885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:12.544840 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:14:12.545015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:12.544970 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:14:12.545090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:12.545018 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:14:13.478101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:13.478067 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" event={"ID":"345f5596-f75f-4aa1-adc4-d97f16ffa1a0","Type":"ContainerStarted","Data":"c36746517a2d58b39e1afad171c5503974227c8c7aefe38d004de031be7e156c"} Oct 29 22:14:13.478519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:13.478121 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" event={"ID":"345f5596-f75f-4aa1-adc4-d97f16ffa1a0","Type":"ContainerStarted","Data":"04ebe3583d11551286f3cbc9eaa0e1276c5a4af208d68728160b0dcce07350ba"} Oct 29 22:14:13.552919 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:13.552851 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" podStartSLOduration=2.552834354 podStartE2EDuration="2.552834354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:13.5350961 +0000 UTC m=+1919.009684519" watchObservedRunningTime="2026-10-29 22:14:13.552834354 +0000 UTC m=+1919.027422780" Oct 29 22:14:13.976240 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:13.966253 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:13.976240 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:13.975944 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:14.039456 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:14.039412 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:14.488154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:14.488118 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:14.489308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:14.489278 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:14:15.538102 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:14:15.538065 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:14:15.660240 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:15.660206 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:16.023727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:16.023659 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:16.350923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:16.346563 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:16.522151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:16.522111 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:18.199336 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:18.199298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:18.789030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:18.788997 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:18.970982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:18.969259 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:19.653078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:19.653041 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:19.668731 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:19.668702 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:19.964991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:19.964716 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:20.547554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:20.547522 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:14:20.837974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:20.837892 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:21.336667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:21.336632 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:21.785188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:21.785081 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:21.967823 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:21.967784 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:23.558456 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:23.558418 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:25.300799 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:25.300759 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:25.483352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:25.483317 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:26.961617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:26.961251 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp"] Oct 29 22:14:26.967533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:26.967508 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:26.976024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:26.975993 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6"] Oct 29 22:14:26.981350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:26.981323 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.010659 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.010628 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp"] Oct 29 22:14:27.018055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.018021 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6"] Oct 29 22:14:27.025701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.025656 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:27.071128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071027 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.071420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071133 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgknw\" (UniqueName: \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-kube-api-access-vgknw\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.071420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071161 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.071420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071213 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.071420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071239 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.071420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071361 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.071798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071443 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.071798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071469 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.071798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071494 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55eb0002-813c-4f3f-9951-4bbe18a8802c-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.071798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071535 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.071798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071568 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.071798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071627 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrp67\" (UniqueName: \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-kube-api-access-qrp67\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.071798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071646 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.071798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071664 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.071798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071690 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.071798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071792 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.072453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071816 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.072453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.071856 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.173717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.173677 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.173717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.173720 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.174077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.173761 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.174077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.173803 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.174077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.173834 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgknw\" (UniqueName: \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-kube-api-access-vgknw\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.174077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.173859 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.174077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.173948 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.174077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.173977 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.174077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174070 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.174465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174155 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.174465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174185 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.174465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174213 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55eb0002-813c-4f3f-9951-4bbe18a8802c-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.174465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174251 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.174465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174284 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.174465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174342 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrp67\" (UniqueName: \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-kube-api-access-qrp67\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.174465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174396 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.174465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174432 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.174465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174459 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.175013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174635 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.175013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174864 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.175013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174960 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.175013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.174968 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-credential-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.175278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.175136 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.175278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.175254 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55eb0002-813c-4f3f-9951-4bbe18a8802c-istiod-ca-cert\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.175450 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.175260 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.175450 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.175401 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-certs\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.175563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.175470 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-socket\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.175647 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.175622 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-data\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.178065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.178036 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.178065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.178052 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.178240 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.178051 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-podinfo\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.178361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.178338 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-envoy\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.204847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.204807 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.205051 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.204949 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgknw\" (UniqueName: \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-kube-api-access-vgknw\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.205051 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.204960 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-token\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.205235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.205215 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrp67\" (UniqueName: \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-kube-api-access-qrp67\") pod \"e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.207046 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.206939 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:27.301480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.300962 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:27.316260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.316222 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:27.865708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.865673 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:27.929015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.928841 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6"] Oct 29 22:14:27.930089 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:14:27.930058 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85eb5f26_c0eb_47a8_b260_ca2eed4ad3b2.slice/crio-23585ef5ef6b64fd92295bc95193f23c5059955237bc7673e1c78f22ff5b231e WatchSource:0}: Error finding container 23585ef5ef6b64fd92295bc95193f23c5059955237bc7673e1c78f22ff5b231e: Status 404 returned error can't find the container with id 23585ef5ef6b64fd92295bc95193f23c5059955237bc7673e1c78f22ff5b231e Oct 29 22:14:27.931818 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:14:27.931792 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55eb0002_813c_4f3f_9951_4bbe18a8802c.slice/crio-646cc1b0939622087444675a43b2e5132e76acf2fd43fc05802aa6d21d3901f3 WatchSource:0}: Error finding container 646cc1b0939622087444675a43b2e5132e76acf2fd43fc05802aa6d21d3901f3: Status 404 returned error can't find the container with id 646cc1b0939622087444675a43b2e5132e76acf2fd43fc05802aa6d21d3901f3 Oct 29 22:14:27.933487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.933453 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:14:27.933602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.933531 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:14:27.933602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.933574 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:14:27.934218 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.934197 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp"] Oct 29 22:14:27.934962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.934881 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:14:27.935056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.935011 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:14:27.935113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:27.935056 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:14:28.701599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:28.701561 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" event={"ID":"55eb0002-813c-4f3f-9951-4bbe18a8802c","Type":"ContainerStarted","Data":"3c7bd49f2ec0cf5d178a70a9a867a6bf91b041c7acda8b3c666918d7e9fb6e94"} Oct 29 22:14:28.702117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:28.701629 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" event={"ID":"55eb0002-813c-4f3f-9951-4bbe18a8802c","Type":"ContainerStarted","Data":"646cc1b0939622087444675a43b2e5132e76acf2fd43fc05802aa6d21d3901f3"} Oct 29 22:14:28.702117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:28.701652 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" event={"ID":"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2","Type":"ContainerStarted","Data":"2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af"} Oct 29 22:14:28.702117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:28.701675 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" event={"ID":"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2","Type":"ContainerStarted","Data":"23585ef5ef6b64fd92295bc95193f23c5059955237bc7673e1c78f22ff5b231e"} Oct 29 22:14:28.780929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:28.780268 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" podStartSLOduration=2.780241095 podStartE2EDuration="2.780241095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:28.768463407 +0000 UTC m=+1934.243051829" watchObservedRunningTime="2026-10-29 22:14:28.780241095 +0000 UTC m=+1934.254829516" Oct 29 22:14:28.950053 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:28.920700 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" podStartSLOduration=2.920677777 podStartE2EDuration="2.920677777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:28.909881438 +0000 UTC m=+1934.384469857" watchObservedRunningTime="2026-10-29 22:14:28.920677777 +0000 UTC m=+1934.395266196" Oct 29 22:14:29.306088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:29.306052 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:29.307343 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:29.307318 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 22:14:29.307416 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:29.307368 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 22:14:29.367308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:29.367260 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:29.367886 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:29.367795 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 29 22:14:29.367886 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:29.367861 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 29 22:14:30.059517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:30.059479 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:30.307183 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:30.306925 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 22:14:30.307183 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:30.306990 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 22:14:30.362929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:30.362580 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 29 22:14:30.362929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:30.362649 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 29 22:14:31.306933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:31.306500 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 22:14:31.306933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:31.306574 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 22:14:31.359534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:31.359498 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 29 22:14:31.359830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:31.359801 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 29 22:14:31.977928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:31.961962 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th"] Oct 29 22:14:32.017423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.016312 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th"] Oct 29 22:14:32.017423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.016438 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.081932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.072014 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:32.097257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.097214 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-675bf4-serving-cert\"" Oct 29 22:14:32.249009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.248567 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f169249-a54d-45cc-a71e-52a5bd54d81d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.249009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.248715 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.249009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.248823 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.249009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.248851 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvft2\" (UniqueName: \"kubernetes.io/projected/9f169249-a54d-45cc-a71e-52a5bd54d81d-kube-api-access-xvft2\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.287877 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.280399 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:32.306776 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.306738 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 22:14:32.306977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.306838 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 22:14:32.386929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.379728 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 29 22:14:32.386929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.379787 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 29 22:14:32.386929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.381015 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f169249-a54d-45cc-a71e-52a5bd54d81d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.386929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.381127 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.386929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.381218 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.386929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.381244 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvft2\" (UniqueName: \"kubernetes.io/projected/9f169249-a54d-45cc-a71e-52a5bd54d81d-kube-api-access-xvft2\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.386929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.384604 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f169249-a54d-45cc-a71e-52a5bd54d81d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.394684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.394547 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.395733 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.395674 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.405869 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.405844 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvft2\" (UniqueName: \"kubernetes.io/projected/9f169249-a54d-45cc-a71e-52a5bd54d81d-kube-api-access-xvft2\") pod \"maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.490198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.490157 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:32.924960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:32.920189 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th"] Oct 29 22:14:32.981631 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:14:32.981593 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f169249_a54d_45cc_a71e_52a5bd54d81d.slice/crio-4070921eeeaa6457753e185dbee727bfca8abdb49c65f3984ae6a535f86c80c0 WatchSource:0}: Error finding container 4070921eeeaa6457753e185dbee727bfca8abdb49c65f3984ae6a535f86c80c0: Status 404 returned error can't find the container with id 4070921eeeaa6457753e185dbee727bfca8abdb49c65f3984ae6a535f86c80c0 Oct 29 22:14:33.308421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:33.308319 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 22:14:33.308421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:33.308384 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 22:14:33.360471 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:33.360434 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 29 22:14:33.360793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:33.360763 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 29 22:14:33.921187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:33.907000 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" event={"ID":"9f169249-a54d-45cc-a71e-52a5bd54d81d","Type":"ContainerStarted","Data":"4070921eeeaa6457753e185dbee727bfca8abdb49c65f3984ae6a535f86c80c0"} Oct 29 22:14:34.306206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:34.306116 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 22:14:34.306206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:34.306175 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 22:14:34.362926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:34.362251 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 29 22:14:34.362926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:34.362312 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 29 22:14:34.518925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:34.514544 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:34.920124 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:34.920067 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" event={"ID":"9f169249-a54d-45cc-a71e-52a5bd54d81d","Type":"ContainerStarted","Data":"2ef6aba0a567a7d7cdc2df524ffe2f0fea1917564d06827b874b3191c09a43f6"} Oct 29 22:14:34.920298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:34.920182 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:34.984029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:34.983968 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" podStartSLOduration=3.284299195 podStartE2EDuration="3.983951614s" totalImagesPullingTime="699.652419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:34.978683353 +0000 UTC m=+1940.453271784" watchObservedRunningTime="2026-10-29 22:14:34.983951614 +0000 UTC m=+1940.458540033" Oct 29 22:14:35.306873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.306774 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 29 22:14:35.306873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.306841 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 29 22:14:35.362586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.359156 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 29 22:14:35.362586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.359234 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 29 22:14:35.599583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.599166 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct"] Oct 29 22:14:35.629928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.620599 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct"] Oct 29 22:14:35.629928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.620759 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.674364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.661645 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-76c1b7\"/\"openshift-service-ca.crt\"" Oct 29 22:14:35.674364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.661744 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-76c1b7\"/\"default-dockercfg-lk8wf\"" Oct 29 22:14:35.674364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.661813 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-76c1b7\"/\"kube-root-ca.crt\"" Oct 29 22:14:35.674364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.661952 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-76c1b7\"/\"auth-test-model-76c1b7-kserve-self-signed-certs\"" Oct 29 22:14:35.725032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.717862 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf26737-8758-4654-8213-3e0d7db85ee7-tls-certs\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.725032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.718117 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-dshm\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.725032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.718227 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-home\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.725032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.718327 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-model-cache\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.725032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.718483 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-tmp-dir\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.725032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.718636 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnp4s\" (UniqueName: \"kubernetes.io/projected/edf26737-8758-4654-8213-3e0d7db85ee7-kube-api-access-qnp4s\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.760480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.760445 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:35.826937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.826847 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-model-cache\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.827133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.827005 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-tmp-dir\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.851975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.827183 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnp4s\" (UniqueName: \"kubernetes.io/projected/edf26737-8758-4654-8213-3e0d7db85ee7-kube-api-access-qnp4s\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.851975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.827308 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf26737-8758-4654-8213-3e0d7db85ee7-tls-certs\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.851975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.827381 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-dshm\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.851975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.827432 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-home\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.851975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.827939 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-home\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.851975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.828456 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-model-cache\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:35.851975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:35.828838 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-tmp-dir\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:36.126422 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.126345 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf26737-8758-4654-8213-3e0d7db85ee7-tls-certs\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:36.138281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.138247 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-dshm\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:36.139261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.139197 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnp4s\" (UniqueName: \"kubernetes.io/projected/edf26737-8758-4654-8213-3e0d7db85ee7-kube-api-access-qnp4s\") pod \"auth-test-model-76c1b7-kserve-668977b7b5-6x9ct\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:36.317701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.316595 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:36.534667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.534634 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:36.536067 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.535596 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:36.537536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.537475 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:14:36.593931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.592488 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:36.593931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.593767 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:36.595433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.595285 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:14:36.755674 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.755314 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct"] Oct 29 22:14:36.756088 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:14:36.755828 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedf26737_8758_4654_8213_3e0d7db85ee7.slice/crio-777892fc7143a891d8350e0fe5d92f0e83557e8d5e97b35e0934238124450b17 WatchSource:0}: Error finding container 777892fc7143a891d8350e0fe5d92f0e83557e8d5e97b35e0934238124450b17: Status 404 returned error can't find the container with id 777892fc7143a891d8350e0fe5d92f0e83557e8d5e97b35e0934238124450b17 Oct 29 22:14:36.798358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:36.798326 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:37.082779 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:37.082737 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" event={"ID":"edf26737-8758-4654-8213-3e0d7db85ee7","Type":"ContainerStarted","Data":"a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d"} Oct 29 22:14:37.082779 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:37.082782 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" event={"ID":"edf26737-8758-4654-8213-3e0d7db85ee7","Type":"ContainerStarted","Data":"777892fc7143a891d8350e0fe5d92f0e83557e8d5e97b35e0934238124450b17"} Oct 29 22:14:37.128921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:37.128856 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" podStartSLOduration=2.128841455 podStartE2EDuration="2.128841455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:37.124193916 +0000 UTC m=+1942.598782336" watchObservedRunningTime="2026-10-29 22:14:37.128841455 +0000 UTC m=+1942.603429873" Oct 29 22:14:37.317884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:37.317827 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:38.532445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:38.528639 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:40.282886 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:40.281863 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:41.078018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:41.077983 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:14:41.978580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:41.978540 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:42.191062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:42.191026 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:42.376304 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:42.376260 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:43.873384 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:43.869146 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:44.770012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:44.769972 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:14:45.530856 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:14:45.530812 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:14:45.583321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.583260 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs"] Oct 29 22:14:45.591119 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.591079 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.615091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.615051 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs"] Oct 29 22:14:45.641163 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.641044 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-971004\"/\"openshift-service-ca.crt\"" Oct 29 22:14:45.641163 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.641086 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-971004\"/\"kube-root-ca.crt\"" Oct 29 22:14:45.653182 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.653155 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-971004\"/\"default-dockercfg-vvhhw\"" Oct 29 22:14:45.653394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.653168 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-971004\"/\"autores-model-971004-kserve-self-signed-certs\"" Oct 29 22:14:45.684481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.684441 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-model-cache\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.684681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.684580 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-dshm\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.684681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.684615 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-home\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.684681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.684673 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-tmp-dir\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.684854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.684730 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9825f391-5b6d-4575-820f-62febeeb352b-tls-certs\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.684854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.684771 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xl89\" (UniqueName: \"kubernetes.io/projected/9825f391-5b6d-4575-820f-62febeeb352b-kube-api-access-4xl89\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.709216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.709181 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:45.786118 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.786035 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-model-cache\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.786275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.786144 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-dshm\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.786275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.786198 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-home\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.786458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.786325 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-tmp-dir\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.786458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.786450 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9825f391-5b6d-4575-820f-62febeeb352b-tls-certs\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.786595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.786510 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-model-cache\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.786595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.786523 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xl89\" (UniqueName: \"kubernetes.io/projected/9825f391-5b6d-4575-820f-62febeeb352b-kube-api-access-4xl89\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.786716 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.786656 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-home\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.786820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.786801 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-tmp-dir\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.789311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.789236 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-dshm\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.789443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.789365 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9825f391-5b6d-4575-820f-62febeeb352b-tls-certs\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.812085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.812051 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xl89\" (UniqueName: \"kubernetes.io/projected/9825f391-5b6d-4575-820f-62febeeb352b-kube-api-access-4xl89\") pod \"autores-model-971004-kserve-65c6bc8dfc-wpxrs\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.953500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.953425 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:45.956990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:45.956364 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:46.327008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:46.326105 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:46.468932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:46.465487 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:46.904874 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:14:46.904841 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9825f391_5b6d_4575_820f_62febeeb352b.slice/crio-5bdec6dabcbe186de165d3797410db2f1f70a4233dfdf4f8805ff995b2751b45 WatchSource:0}: Error finding container 5bdec6dabcbe186de165d3797410db2f1f70a4233dfdf4f8805ff995b2751b45: Status 404 returned error can't find the container with id 5bdec6dabcbe186de165d3797410db2f1f70a4233dfdf4f8805ff995b2751b45 Oct 29 22:14:46.906967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:46.906937 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs"] Oct 29 22:14:47.392199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:47.392166 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:14:47.412957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:47.412421 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" event={"ID":"9825f391-5b6d-4575-820f-62febeeb352b","Type":"ContainerStarted","Data":"0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d"} Oct 29 22:14:47.412957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:47.412470 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" event={"ID":"9825f391-5b6d-4575-820f-62febeeb352b","Type":"ContainerStarted","Data":"5bdec6dabcbe186de165d3797410db2f1f70a4233dfdf4f8805ff995b2751b45"} Oct 29 22:14:47.503847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:47.502834 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" podStartSLOduration=2.5028178949999997 podStartE2EDuration="2.502817895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:47.497119636 +0000 UTC m=+1952.971708055" watchObservedRunningTime="2026-10-29 22:14:47.502817895 +0000 UTC m=+1952.977406314" Oct 29 22:14:47.957374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:47.954976 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:48.013926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:48.010475 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:49.542702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.542662 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2"] Oct 29 22:14:49.547494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.547469 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.574932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.574312 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-8d569e-serving-cert\"" Oct 29 22:14:49.579594 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.579562 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2"] Oct 29 22:14:49.663815 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.663779 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.664029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.663835 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89a0aba-122c-49b4-95c5-c38bd87cbda9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.664096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.664037 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.664158 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.664144 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8brl\" (UniqueName: \"kubernetes.io/projected/b89a0aba-122c-49b4-95c5-c38bd87cbda9-kube-api-access-q8brl\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.682883 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.682849 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:49.766588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.766534 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89a0aba-122c-49b4-95c5-c38bd87cbda9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.766899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.766680 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.766899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.766751 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8brl\" (UniqueName: \"kubernetes.io/projected/b89a0aba-122c-49b4-95c5-c38bd87cbda9-kube-api-access-q8brl\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.767133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.767105 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.768886 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.768855 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89a0aba-122c-49b4-95c5-c38bd87cbda9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.771667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.771337 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.780213 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.780166 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.794439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.794238 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8brl\" (UniqueName: \"kubernetes.io/projected/b89a0aba-122c-49b4-95c5-c38bd87cbda9-kube-api-access-q8brl\") pod \"maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:49.856648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.856603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:49.864682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:49.864650 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:50.022325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:50.022278 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:50.185233 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:14:50.185198 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89a0aba_122c_49b4_95c5_c38bd87cbda9.slice/crio-2be8f40931d7f77317f96925045faa702235a9b3a2071eed920ef9da1d69d843 WatchSource:0}: Error finding container 2be8f40931d7f77317f96925045faa702235a9b3a2071eed920ef9da1d69d843: Status 404 returned error can't find the container with id 2be8f40931d7f77317f96925045faa702235a9b3a2071eed920ef9da1d69d843 Oct 29 22:14:50.189138 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:50.189111 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2"] Oct 29 22:14:50.477987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:50.475712 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" event={"ID":"b89a0aba-122c-49b4-95c5-c38bd87cbda9","Type":"ContainerStarted","Data":"2be8f40931d7f77317f96925045faa702235a9b3a2071eed920ef9da1d69d843"} Oct 29 22:14:51.491591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:51.491539 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" event={"ID":"b89a0aba-122c-49b4-95c5-c38bd87cbda9","Type":"ContainerStarted","Data":"b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69"} Oct 29 22:14:51.492080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:51.491611 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:51.546077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:51.543305 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" podStartSLOduration=2.143645445 podStartE2EDuration="2.54328461s" totalImagesPullingTime="399.639165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:51.53844303 +0000 UTC m=+1957.013031461" watchObservedRunningTime="2026-10-29 22:14:51.54328461 +0000 UTC m=+1957.017873030" Oct 29 22:14:51.572958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:51.572793 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:53.253588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:53.253551 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:53.917315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:53.917277 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6"] Oct 29 22:14:53.922018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:53.921991 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:53.946235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:53.946206 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6"] Oct 29 22:14:53.953742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:53.953711 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-15154b\"/\"openshift-service-ca.crt\"" Oct 29 22:14:53.953924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:53.953746 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-15154b\"/\"kube-root-ca.crt\"" Oct 29 22:14:53.953924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:53.953828 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-15154b\"/\"auth-test-model-15154b-kserve-self-signed-certs\"" Oct 29 22:14:53.954018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:53.953980 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-15154b\"/\"default-dockercfg-5r7sv\"" Oct 29 22:14:54.030840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.030797 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:54.052665 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.052627 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-model-cache\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.052865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.052683 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-home\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.052865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.052760 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-dshm\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.053095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.053070 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-tmp-dir\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.053250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.053234 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c25e40-1aee-4a0c-aba8-ec071797192e-tls-certs\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.053299 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.053269 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thbw\" (UniqueName: \"kubernetes.io/projected/d2c25e40-1aee-4a0c-aba8-ec071797192e-kube-api-access-2thbw\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.176977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.167512 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c25e40-1aee-4a0c-aba8-ec071797192e-tls-certs\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.176977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.167578 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thbw\" (UniqueName: \"kubernetes.io/projected/d2c25e40-1aee-4a0c-aba8-ec071797192e-kube-api-access-2thbw\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.176977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.167633 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-model-cache\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.176977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.167690 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-home\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.176977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.167737 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-dshm\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.176977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.168101 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-tmp-dir\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.176977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.168647 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-tmp-dir\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.176977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.169174 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-model-cache\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.176977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.169803 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-home\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.222952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.202638 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c25e40-1aee-4a0c-aba8-ec071797192e-tls-certs\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.222952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.203932 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thbw\" (UniqueName: \"kubernetes.io/projected/d2c25e40-1aee-4a0c-aba8-ec071797192e-kube-api-access-2thbw\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.222952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.211526 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-dshm\") pod \"auth-test-model-15154b-kserve-7f6bcdb45-gfpj6\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.253927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.249224 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:54.701961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:54.701883 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6"] Oct 29 22:14:54.704329 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:14:54.704299 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2c25e40_1aee_4a0c_aba8_ec071797192e.slice/crio-7589eafde4a4d6c429211bf1ed645425c0844e02ff314d7121c2a2818fc90615 WatchSource:0}: Error finding container 7589eafde4a4d6c429211bf1ed645425c0844e02ff314d7121c2a2818fc90615: Status 404 returned error can't find the container with id 7589eafde4a4d6c429211bf1ed645425c0844e02ff314d7121c2a2818fc90615 Oct 29 22:14:55.132217 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:55.132176 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:55.530011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:55.529921 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" event={"ID":"d2c25e40-1aee-4a0c-aba8-ec071797192e","Type":"ContainerStarted","Data":"09d2dacb89b92d40f3d053a9d1a678fe38cdc7c5148c5bf9ea03598c32da2d0b"} Oct 29 22:14:55.530011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:55.529974 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" event={"ID":"d2c25e40-1aee-4a0c-aba8-ec071797192e","Type":"ContainerStarted","Data":"7589eafde4a4d6c429211bf1ed645425c0844e02ff314d7121c2a2818fc90615"} Oct 29 22:14:55.599011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:55.598960 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" podStartSLOduration=2.598945595 podStartE2EDuration="2.598945595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:55.59433682 +0000 UTC m=+1961.068925239" watchObservedRunningTime="2026-10-29 22:14:55.598945595 +0000 UTC m=+1961.073534013" Oct 29 22:14:55.954242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:55.954202 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:55.967248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:55.967219 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:56.249962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:56.249864 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:14:56.558742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:56.558651 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:14:56.841122 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:56.841081 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:57.506217 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:57.506186 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:14:58.696531 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:14:58.696492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:00.170820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.170754 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc"] Oct 29 22:15:00.177410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.177362 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22"] Oct 29 22:15:00.178750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.178718 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" Oct 29 22:15:00.183661 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.183602 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" Oct 29 22:15:00.186491 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.186461 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr"] Oct 29 22:15:00.208101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.202042 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2dcl\" (UniqueName: \"kubernetes.io/projected/6e683892-5640-48ab-8737-b02f21a939a1-kube-api-access-b2dcl\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc\" (UID: \"6e683892-5640-48ab-8737-b02f21a939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" Oct 29 22:15:00.208101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.202207 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e683892-5640-48ab-8737-b02f21a939a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc\" (UID: \"6e683892-5640-48ab-8737-b02f21a939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" Oct 29 22:15:00.208101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.205082 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" Oct 29 22:15:00.219969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.214980 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc"] Oct 29 22:15:00.223655 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.222767 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv"] Oct 29 22:15:00.233440 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.232535 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" Oct 29 22:15:00.243020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.238424 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22"] Oct 29 22:15:00.263304 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.253739 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2"] Oct 29 22:15:00.263304 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.261513 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" Oct 29 22:15:00.264915 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.264874 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr"] Oct 29 22:15:00.275216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.274881 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt"] Oct 29 22:15:00.282070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.281237 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:15:00.285187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.285161 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" Oct 29 22:15:00.290612 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.290585 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx"] Oct 29 22:15:00.304750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.304717 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" Oct 29 22:15:00.306172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.306134 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2dcl\" (UniqueName: \"kubernetes.io/projected/6e683892-5640-48ab-8737-b02f21a939a1-kube-api-access-b2dcl\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc\" (UID: \"6e683892-5640-48ab-8737-b02f21a939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" Oct 29 22:15:00.306287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.306224 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610a2b6f-c986-4c94-834c-b17f655caba7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr\" (UID: \"610a2b6f-c986-4c94-834c-b17f655caba7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" Oct 29 22:15:00.306287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.306274 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgnpp\" (UniqueName: \"kubernetes.io/projected/f7fa4c80-4284-41f6-8dde-d761102e4138-kube-api-access-hgnpp\") pod \"maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv\" (UID: \"f7fa4c80-4284-41f6-8dde-d761102e4138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" Oct 29 22:15:00.306405 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.306336 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e683892-5640-48ab-8737-b02f21a939a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc\" (UID: \"6e683892-5640-48ab-8737-b02f21a939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" Oct 29 22:15:00.306478 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.306409 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skkmt\" (UniqueName: \"kubernetes.io/projected/610a2b6f-c986-4c94-834c-b17f655caba7-kube-api-access-skkmt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr\" (UID: \"610a2b6f-c986-4c94-834c-b17f655caba7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" Oct 29 22:15:00.306478 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.306449 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2\" (UID: \"41f63e45-db88-4f25-aeb3-7f8cb5cec37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" Oct 29 22:15:00.306583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.306509 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa4c80-4284-41f6-8dde-d761102e4138-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv\" (UID: \"f7fa4c80-4284-41f6-8dde-d761102e4138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" Oct 29 22:15:00.306583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.306546 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbsdd\" (UniqueName: \"kubernetes.io/projected/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-kube-api-access-vbsdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2\" (UID: \"41f63e45-db88-4f25-aeb3-7f8cb5cec37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" Oct 29 22:15:00.306696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.306603 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gt7b\" (UniqueName: \"kubernetes.io/projected/30391eb2-4ecd-4a26-9e63-27d1284a338c-kube-api-access-7gt7b\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22\" (UID: \"30391eb2-4ecd-4a26-9e63-27d1284a338c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" Oct 29 22:15:00.306696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.306667 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30391eb2-4ecd-4a26-9e63-27d1284a338c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22\" (UID: \"30391eb2-4ecd-4a26-9e63-27d1284a338c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" Oct 29 22:15:00.309281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.307884 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e683892-5640-48ab-8737-b02f21a939a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc\" (UID: \"6e683892-5640-48ab-8737-b02f21a939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" Oct 29 22:15:00.311252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.311224 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2"] Oct 29 22:15:00.356080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.353498 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2dcl\" (UniqueName: \"kubernetes.io/projected/6e683892-5640-48ab-8737-b02f21a939a1-kube-api-access-b2dcl\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc\" (UID: \"6e683892-5640-48ab-8737-b02f21a939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" Oct 29 22:15:00.356080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.355956 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv"] Oct 29 22:15:00.362633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.362048 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt"] Oct 29 22:15:00.371651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.369363 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx"] Oct 29 22:15:00.408081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408040 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1fb38f-4266-4290-ae59-94dc484c750f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx\" (UID: \"ac1fb38f-4266-4290-ae59-94dc484c750f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" Oct 29 22:15:00.408274 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408089 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ea755a-2d21-44a4-9604-564c464c872f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888535-2cdmt\" (UID: \"93ea755a-2d21-44a4-9604-564c464c872f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" Oct 29 22:15:00.408274 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408220 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610a2b6f-c986-4c94-834c-b17f655caba7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr\" (UID: \"610a2b6f-c986-4c94-834c-b17f655caba7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" Oct 29 22:15:00.408274 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408247 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrl8\" (UniqueName: \"kubernetes.io/projected/ac1fb38f-4266-4290-ae59-94dc484c750f-kube-api-access-kmrl8\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx\" (UID: \"ac1fb38f-4266-4290-ae59-94dc484c750f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" Oct 29 22:15:00.408442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408285 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgnpp\" (UniqueName: \"kubernetes.io/projected/f7fa4c80-4284-41f6-8dde-d761102e4138-kube-api-access-hgnpp\") pod \"maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv\" (UID: \"f7fa4c80-4284-41f6-8dde-d761102e4138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" Oct 29 22:15:00.408442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408327 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znmmr\" (UniqueName: \"kubernetes.io/projected/93ea755a-2d21-44a4-9604-564c464c872f-kube-api-access-znmmr\") pod \"maas-api-key-cleanup-29888535-2cdmt\" (UID: \"93ea755a-2d21-44a4-9604-564c464c872f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" Oct 29 22:15:00.408442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408435 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skkmt\" (UniqueName: \"kubernetes.io/projected/610a2b6f-c986-4c94-834c-b17f655caba7-kube-api-access-skkmt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr\" (UID: \"610a2b6f-c986-4c94-834c-b17f655caba7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" Oct 29 22:15:00.408607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408472 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2\" (UID: \"41f63e45-db88-4f25-aeb3-7f8cb5cec37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" Oct 29 22:15:00.408607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408534 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa4c80-4284-41f6-8dde-d761102e4138-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv\" (UID: \"f7fa4c80-4284-41f6-8dde-d761102e4138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" Oct 29 22:15:00.408607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408563 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbsdd\" (UniqueName: \"kubernetes.io/projected/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-kube-api-access-vbsdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2\" (UID: \"41f63e45-db88-4f25-aeb3-7f8cb5cec37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" Oct 29 22:15:00.408768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408615 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gt7b\" (UniqueName: \"kubernetes.io/projected/30391eb2-4ecd-4a26-9e63-27d1284a338c-kube-api-access-7gt7b\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22\" (UID: \"30391eb2-4ecd-4a26-9e63-27d1284a338c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" Oct 29 22:15:00.408768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.408674 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30391eb2-4ecd-4a26-9e63-27d1284a338c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22\" (UID: \"30391eb2-4ecd-4a26-9e63-27d1284a338c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" Oct 29 22:15:00.409533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.409285 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610a2b6f-c986-4c94-834c-b17f655caba7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr\" (UID: \"610a2b6f-c986-4c94-834c-b17f655caba7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" Oct 29 22:15:00.409533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.409497 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30391eb2-4ecd-4a26-9e63-27d1284a338c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22\" (UID: \"30391eb2-4ecd-4a26-9e63-27d1284a338c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" Oct 29 22:15:00.409787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.409761 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2\" (UID: \"41f63e45-db88-4f25-aeb3-7f8cb5cec37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" Oct 29 22:15:00.410279 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.410251 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa4c80-4284-41f6-8dde-d761102e4138-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv\" (UID: \"f7fa4c80-4284-41f6-8dde-d761102e4138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" Oct 29 22:15:00.437457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.437382 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgnpp\" (UniqueName: \"kubernetes.io/projected/f7fa4c80-4284-41f6-8dde-d761102e4138-kube-api-access-hgnpp\") pod \"maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv\" (UID: \"f7fa4c80-4284-41f6-8dde-d761102e4138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" Oct 29 22:15:00.438047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.438020 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gt7b\" (UniqueName: \"kubernetes.io/projected/30391eb2-4ecd-4a26-9e63-27d1284a338c-kube-api-access-7gt7b\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22\" (UID: \"30391eb2-4ecd-4a26-9e63-27d1284a338c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" Oct 29 22:15:00.438153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.438075 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbsdd\" (UniqueName: \"kubernetes.io/projected/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-kube-api-access-vbsdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2\" (UID: \"41f63e45-db88-4f25-aeb3-7f8cb5cec37e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" Oct 29 22:15:00.438216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.438170 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skkmt\" (UniqueName: \"kubernetes.io/projected/610a2b6f-c986-4c94-834c-b17f655caba7-kube-api-access-skkmt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr\" (UID: \"610a2b6f-c986-4c94-834c-b17f655caba7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" Oct 29 22:15:00.495262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.495227 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:00.510749 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.510702 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1fb38f-4266-4290-ae59-94dc484c750f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx\" (UID: \"ac1fb38f-4266-4290-ae59-94dc484c750f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" Oct 29 22:15:00.510898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.510761 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ea755a-2d21-44a4-9604-564c464c872f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888535-2cdmt\" (UID: \"93ea755a-2d21-44a4-9604-564c464c872f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" Oct 29 22:15:00.510898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.510843 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrl8\" (UniqueName: \"kubernetes.io/projected/ac1fb38f-4266-4290-ae59-94dc484c750f-kube-api-access-kmrl8\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx\" (UID: \"ac1fb38f-4266-4290-ae59-94dc484c750f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" Oct 29 22:15:00.510898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.510885 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znmmr\" (UniqueName: \"kubernetes.io/projected/93ea755a-2d21-44a4-9604-564c464c872f-kube-api-access-znmmr\") pod \"maas-api-key-cleanup-29888535-2cdmt\" (UID: \"93ea755a-2d21-44a4-9604-564c464c872f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" Oct 29 22:15:00.511517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.511495 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1fb38f-4266-4290-ae59-94dc484c750f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx\" (UID: \"ac1fb38f-4266-4290-ae59-94dc484c750f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" Oct 29 22:15:00.511609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.511521 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ea755a-2d21-44a4-9604-564c464c872f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888535-2cdmt\" (UID: \"93ea755a-2d21-44a4-9604-564c464c872f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" Oct 29 22:15:00.535515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.535481 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" Oct 29 22:15:00.538580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.538547 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znmmr\" (UniqueName: \"kubernetes.io/projected/93ea755a-2d21-44a4-9604-564c464c872f-kube-api-access-znmmr\") pod \"maas-api-key-cleanup-29888535-2cdmt\" (UID: \"93ea755a-2d21-44a4-9604-564c464c872f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" Oct 29 22:15:00.538753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.538547 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrl8\" (UniqueName: \"kubernetes.io/projected/ac1fb38f-4266-4290-ae59-94dc484c750f-kube-api-access-kmrl8\") pod \"maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx\" (UID: \"ac1fb38f-4266-4290-ae59-94dc484c750f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" Oct 29 22:15:00.550008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.549969 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" Oct 29 22:15:00.578006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.577975 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" Oct 29 22:15:00.625205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.624247 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" Oct 29 22:15:00.638042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.637733 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" Oct 29 22:15:00.638042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.637883 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" Oct 29 22:15:00.638042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.637978 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" Oct 29 22:15:00.702383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.702237 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:00.996235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:00.996198 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc"] Oct 29 22:15:01.000473 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:15:01.000439 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e683892_5640_48ab_8737_b02f21a939a1.slice/crio-dfe2f5791d242c01e765bffda07dfde714fe666d322722c689b6fff23211c4a5 WatchSource:0}: Error finding container dfe2f5791d242c01e765bffda07dfde714fe666d322722c689b6fff23211c4a5: Status 404 returned error can't find the container with id dfe2f5791d242c01e765bffda07dfde714fe666d322722c689b6fff23211c4a5 Oct 29 22:15:01.213300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.213262 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22"] Oct 29 22:15:01.234839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.234789 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx"] Oct 29 22:15:01.240863 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:15:01.240801 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1fb38f_4266_4290_ae59_94dc484c750f.slice/crio-9576b39bd9a36c40e4ea3822e1a8d7df19027b4150b5010dc0e60942e5918055 WatchSource:0}: Error finding container 9576b39bd9a36c40e4ea3822e1a8d7df19027b4150b5010dc0e60942e5918055: Status 404 returned error can't find the container with id 9576b39bd9a36c40e4ea3822e1a8d7df19027b4150b5010dc0e60942e5918055 Oct 29 22:15:01.398720 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.398635 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr"] Oct 29 22:15:01.401951 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:15:01.401873 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ea755a_2d21_44a4_9604_564c464c872f.slice/crio-dbe850b14279d1ada4f83f4f4b90c8a68567f62ee72c493aa0d7333bf1a7c61e WatchSource:0}: Error finding container dbe850b14279d1ada4f83f4f4b90c8a68567f62ee72c493aa0d7333bf1a7c61e: Status 404 returned error can't find the container with id dbe850b14279d1ada4f83f4f4b90c8a68567f62ee72c493aa0d7333bf1a7c61e Oct 29 22:15:01.407324 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:15:01.407278 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f63e45_db88_4f25_aeb3_7f8cb5cec37e.slice/crio-28d38e65983fa4d0b03f4e8aa527badac0cd95fd89d5bd0d11d05f481a95a7af WatchSource:0}: Error finding container 28d38e65983fa4d0b03f4e8aa527badac0cd95fd89d5bd0d11d05f481a95a7af: Status 404 returned error can't find the container with id 28d38e65983fa4d0b03f4e8aa527badac0cd95fd89d5bd0d11d05f481a95a7af Oct 29 22:15:01.408770 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:15:01.408721 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fa4c80_4284_41f6_8dde_d761102e4138.slice/crio-0779cbb150a535f3d2af7bbef0ccd20427001e8b568cd898d59bd6043bcf0194 WatchSource:0}: Error finding container 0779cbb150a535f3d2af7bbef0ccd20427001e8b568cd898d59bd6043bcf0194: Status 404 returned error can't find the container with id 0779cbb150a535f3d2af7bbef0ccd20427001e8b568cd898d59bd6043bcf0194 Oct 29 22:15:01.420089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.420062 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv"] Oct 29 22:15:01.433607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.433586 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2"] Oct 29 22:15:01.434458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.434434 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt"] Oct 29 22:15:01.671313 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.665752 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" event={"ID":"f7fa4c80-4284-41f6-8dde-d761102e4138","Type":"ContainerStarted","Data":"0779cbb150a535f3d2af7bbef0ccd20427001e8b568cd898d59bd6043bcf0194"} Oct 29 22:15:01.686548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.686505 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" event={"ID":"93ea755a-2d21-44a4-9604-564c464c872f","Type":"ContainerStarted","Data":"dbe850b14279d1ada4f83f4f4b90c8a68567f62ee72c493aa0d7333bf1a7c61e"} Oct 29 22:15:01.695157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.689598 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" event={"ID":"6e683892-5640-48ab-8737-b02f21a939a1","Type":"ContainerStarted","Data":"202fc05bd7297a5df6b2461a3981ab0a0fc1e880745d3041f7c4e47bc81aa422"} Oct 29 22:15:01.695157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.689662 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" event={"ID":"6e683892-5640-48ab-8737-b02f21a939a1","Type":"ContainerStarted","Data":"dfe2f5791d242c01e765bffda07dfde714fe666d322722c689b6fff23211c4a5"} Oct 29 22:15:01.698060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.698023 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" event={"ID":"ac1fb38f-4266-4290-ae59-94dc484c750f","Type":"ContainerStarted","Data":"9576b39bd9a36c40e4ea3822e1a8d7df19027b4150b5010dc0e60942e5918055"} Oct 29 22:15:01.728611 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.728140 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" event={"ID":"610a2b6f-c986-4c94-834c-b17f655caba7","Type":"ContainerStarted","Data":"d5aacc15aadfa7b9a7ee8c512a837c3dcf47c74d35da02322b46fbbf0fc32152"} Oct 29 22:15:01.730652 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.730252 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" event={"ID":"30391eb2-4ecd-4a26-9e63-27d1284a338c","Type":"ContainerStarted","Data":"07862d1e59fd66de435725dc3a8858cdf9f90f37ed6fa60a34f3bc53c6c673a7"} Oct 29 22:15:01.752531 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.751950 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" podStartSLOduration=1.751930648 podStartE2EDuration="1.751930648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:01.739300719 +0000 UTC m=+1967.213889138" watchObservedRunningTime="2026-10-29 22:15:01.751930648 +0000 UTC m=+1967.226519067" Oct 29 22:15:01.753477 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:01.753401 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" event={"ID":"41f63e45-db88-4f25-aeb3-7f8cb5cec37e","Type":"ContainerStarted","Data":"28d38e65983fa4d0b03f4e8aa527badac0cd95fd89d5bd0d11d05f481a95a7af"} Oct 29 22:15:02.542798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.541756 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:02.764499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.764458 4952 generic.go:356] "Generic (PLEG): container finished" podID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerID="6eb1b16c1bf622187fc663945255a1f98f65b6c2eca0ac17a0eb2e47055ca472" exitCode=60 Oct 29 22:15:02.764680 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.764540 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" event={"ID":"41f63e45-db88-4f25-aeb3-7f8cb5cec37e","Type":"ContainerDied","Data":"6eb1b16c1bf622187fc663945255a1f98f65b6c2eca0ac17a0eb2e47055ca472"} Oct 29 22:15:02.764986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.764961 4952 scope.go:122] "RemoveContainer" containerID="6eb1b16c1bf622187fc663945255a1f98f65b6c2eca0ac17a0eb2e47055ca472" Oct 29 22:15:02.766731 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.766707 4952 generic.go:356] "Generic (PLEG): container finished" podID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerID="32b2c8a9cde0f7db4eb54cf27be9025623f31d9efbca15e6ada65aa1f5cb7455" exitCode=60 Oct 29 22:15:02.766863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.766785 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" event={"ID":"f7fa4c80-4284-41f6-8dde-d761102e4138","Type":"ContainerDied","Data":"32b2c8a9cde0f7db4eb54cf27be9025623f31d9efbca15e6ada65aa1f5cb7455"} Oct 29 22:15:02.767166 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.767133 4952 scope.go:122] "RemoveContainer" containerID="32b2c8a9cde0f7db4eb54cf27be9025623f31d9efbca15e6ada65aa1f5cb7455" Oct 29 22:15:02.769430 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.769407 4952 generic.go:356] "Generic (PLEG): container finished" podID="93ea755a-2d21-44a4-9604-564c464c872f" containerID="3f4c36fce359f6797a26e4b7ce76cc77c690a666214bf250c9d35c30f94ad2bb" exitCode=60 Oct 29 22:15:02.769711 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.769691 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" event={"ID":"93ea755a-2d21-44a4-9604-564c464c872f","Type":"ContainerDied","Data":"3f4c36fce359f6797a26e4b7ce76cc77c690a666214bf250c9d35c30f94ad2bb"} Oct 29 22:15:02.769877 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.769861 4952 scope.go:122] "RemoveContainer" containerID="3f4c36fce359f6797a26e4b7ce76cc77c690a666214bf250c9d35c30f94ad2bb" Oct 29 22:15:02.773844 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.771980 4952 generic.go:356] "Generic (PLEG): container finished" podID="6e683892-5640-48ab-8737-b02f21a939a1" containerID="202fc05bd7297a5df6b2461a3981ab0a0fc1e880745d3041f7c4e47bc81aa422" exitCode=60 Oct 29 22:15:02.773844 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.772048 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" event={"ID":"6e683892-5640-48ab-8737-b02f21a939a1","Type":"ContainerDied","Data":"202fc05bd7297a5df6b2461a3981ab0a0fc1e880745d3041f7c4e47bc81aa422"} Oct 29 22:15:02.773844 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.773094 4952 scope.go:122] "RemoveContainer" containerID="202fc05bd7297a5df6b2461a3981ab0a0fc1e880745d3041f7c4e47bc81aa422" Oct 29 22:15:02.774327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.774130 4952 generic.go:356] "Generic (PLEG): container finished" podID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerID="c4f4e36a22d961487d9ec17a8d3e7fcf2e615ac77b559f8107a70d2b88dbfd48" exitCode=60 Oct 29 22:15:02.774327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.774211 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" event={"ID":"ac1fb38f-4266-4290-ae59-94dc484c750f","Type":"ContainerDied","Data":"c4f4e36a22d961487d9ec17a8d3e7fcf2e615ac77b559f8107a70d2b88dbfd48"} Oct 29 22:15:02.774452 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.774438 4952 scope.go:122] "RemoveContainer" containerID="c4f4e36a22d961487d9ec17a8d3e7fcf2e615ac77b559f8107a70d2b88dbfd48" Oct 29 22:15:02.775879 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.775858 4952 generic.go:356] "Generic (PLEG): container finished" podID="610a2b6f-c986-4c94-834c-b17f655caba7" containerID="0fa392b2c2824420610db0cadfef46113aa8865024f5bab9a598bc9f0ee91bfc" exitCode=60 Oct 29 22:15:02.776009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.775964 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" event={"ID":"610a2b6f-c986-4c94-834c-b17f655caba7","Type":"ContainerDied","Data":"0fa392b2c2824420610db0cadfef46113aa8865024f5bab9a598bc9f0ee91bfc"} Oct 29 22:15:02.776192 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.776173 4952 scope.go:122] "RemoveContainer" containerID="0fa392b2c2824420610db0cadfef46113aa8865024f5bab9a598bc9f0ee91bfc" Oct 29 22:15:02.777772 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.777751 4952 generic.go:356] "Generic (PLEG): container finished" podID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerID="df0a47834ab55ab7fa8c405cde70258faa470df37328a814f09119e213dc3dab" exitCode=60 Oct 29 22:15:02.777880 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.777845 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" event={"ID":"30391eb2-4ecd-4a26-9e63-27d1284a338c","Type":"ContainerDied","Data":"df0a47834ab55ab7fa8c405cde70258faa470df37328a814f09119e213dc3dab"} Oct 29 22:15:02.778047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:02.778032 4952 scope.go:122] "RemoveContainer" containerID="df0a47834ab55ab7fa8c405cde70258faa470df37328a814f09119e213dc3dab" Oct 29 22:15:03.788631 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.788536 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" event={"ID":"41f63e45-db88-4f25-aeb3-7f8cb5cec37e","Type":"ContainerStarted","Data":"f46ebedb74def8853ee101edefb94bb333be2dbaf29292a9730a89bcfce44da6"} Oct 29 22:15:03.790735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.790702 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" event={"ID":"f7fa4c80-4284-41f6-8dde-d761102e4138","Type":"ContainerStarted","Data":"0923260274f691fdc543485a6e4ea3e0b8784d8ad6e4dc8a2c941ef85513330c"} Oct 29 22:15:03.793120 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.793100 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" event={"ID":"93ea755a-2d21-44a4-9604-564c464c872f","Type":"ContainerStarted","Data":"4a906722b19a94a89890111a95066a94e218cb3dbbde37aa47447f9b9ea5b23a"} Oct 29 22:15:03.795082 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.795056 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" event={"ID":"6e683892-5640-48ab-8737-b02f21a939a1","Type":"ContainerStarted","Data":"bbbacf93a141ba4f1cef9093ec6a69f4d5e55186f15680d4cfaca26798ab8625"} Oct 29 22:15:03.796892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.796869 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" event={"ID":"ac1fb38f-4266-4290-ae59-94dc484c750f","Type":"ContainerStarted","Data":"0c7241e16cd138471d64150fac5dd77e32f7fd1861918268489d4e4fe07259fe"} Oct 29 22:15:03.798667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.798648 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" event={"ID":"610a2b6f-c986-4c94-834c-b17f655caba7","Type":"ContainerStarted","Data":"b33a3c262fd7ee07a5ae9e49d9857250d19b9aa820ed0295f552d5587f6df0be"} Oct 29 22:15:03.800786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.800763 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" event={"ID":"30391eb2-4ecd-4a26-9e63-27d1284a338c","Type":"ContainerStarted","Data":"bea779102f0ccf43fd962f903f03fd8b258879b798a7a738fbae7e833e1ce723"} Oct 29 22:15:03.830654 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.830603 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" podStartSLOduration=3.830587623 podStartE2EDuration="3.830587623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:03.828330917 +0000 UTC m=+1969.302919336" watchObservedRunningTime="2026-10-29 22:15:03.830587623 +0000 UTC m=+1969.305176041" Oct 29 22:15:03.871508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.871450 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" podStartSLOduration=3.871436058 podStartE2EDuration="3.871436058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:03.869357247 +0000 UTC m=+1969.343945668" watchObservedRunningTime="2026-10-29 22:15:03.871436058 +0000 UTC m=+1969.346024475" Oct 29 22:15:03.954362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.954300 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" podStartSLOduration=3.95428576 podStartE2EDuration="3.95428576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:03.950416384 +0000 UTC m=+1969.425004801" watchObservedRunningTime="2026-10-29 22:15:03.95428576 +0000 UTC m=+1969.428874177" Oct 29 22:15:03.993301 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:03.993248 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" podStartSLOduration=3.993234474 podStartE2EDuration="3.993234474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:03.990883369 +0000 UTC m=+1969.465471790" watchObservedRunningTime="2026-10-29 22:15:03.993234474 +0000 UTC m=+1969.467822892" Oct 29 22:15:04.090110 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.090063 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" podStartSLOduration=4.090049994 podStartE2EDuration="4.090049994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:04.088783902 +0000 UTC m=+1969.563372317" watchObservedRunningTime="2026-10-29 22:15:04.090049994 +0000 UTC m=+1969.564638412" Oct 29 22:15:04.092303 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.092265 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" podStartSLOduration=4.09225271 podStartE2EDuration="4.09225271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:04.040630914 +0000 UTC m=+1969.515219333" watchObservedRunningTime="2026-10-29 22:15:04.09225271 +0000 UTC m=+1969.566841128" Oct 29 22:15:04.250548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.250514 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:15:04.265772 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.265741 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:15:04.411932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.410115 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:04.813598 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.813506 4952 generic.go:356] "Generic (PLEG): container finished" podID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerID="0c7241e16cd138471d64150fac5dd77e32f7fd1861918268489d4e4fe07259fe" exitCode=60 Oct 29 22:15:04.814040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.813595 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" event={"ID":"ac1fb38f-4266-4290-ae59-94dc484c750f","Type":"ContainerDied","Data":"0c7241e16cd138471d64150fac5dd77e32f7fd1861918268489d4e4fe07259fe"} Oct 29 22:15:04.814040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.813658 4952 scope.go:122] "RemoveContainer" containerID="c4f4e36a22d961487d9ec17a8d3e7fcf2e615ac77b559f8107a70d2b88dbfd48" Oct 29 22:15:04.814040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.814029 4952 scope.go:122] "RemoveContainer" containerID="0c7241e16cd138471d64150fac5dd77e32f7fd1861918268489d4e4fe07259fe" Oct 29 22:15:04.814482 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:04.814456 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx_odh-ai-gateway-infra(ac1fb38f-4266-4290-ae59-94dc484c750f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" Oct 29 22:15:04.816739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.816716 4952 generic.go:356] "Generic (PLEG): container finished" podID="610a2b6f-c986-4c94-834c-b17f655caba7" containerID="b33a3c262fd7ee07a5ae9e49d9857250d19b9aa820ed0295f552d5587f6df0be" exitCode=60 Oct 29 22:15:04.816833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.816782 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" event={"ID":"610a2b6f-c986-4c94-834c-b17f655caba7","Type":"ContainerDied","Data":"b33a3c262fd7ee07a5ae9e49d9857250d19b9aa820ed0295f552d5587f6df0be"} Oct 29 22:15:04.817090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.817070 4952 scope.go:122] "RemoveContainer" containerID="b33a3c262fd7ee07a5ae9e49d9857250d19b9aa820ed0295f552d5587f6df0be" Oct 29 22:15:04.817456 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:04.817436 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr_odh-ai-gateway-infra(610a2b6f-c986-4c94-834c-b17f655caba7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" Oct 29 22:15:04.818686 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.818663 4952 generic.go:356] "Generic (PLEG): container finished" podID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerID="bea779102f0ccf43fd962f903f03fd8b258879b798a7a738fbae7e833e1ce723" exitCode=60 Oct 29 22:15:04.818812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.818745 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" event={"ID":"30391eb2-4ecd-4a26-9e63-27d1284a338c","Type":"ContainerDied","Data":"bea779102f0ccf43fd962f903f03fd8b258879b798a7a738fbae7e833e1ce723"} Oct 29 22:15:04.819041 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.819025 4952 scope.go:122] "RemoveContainer" containerID="bea779102f0ccf43fd962f903f03fd8b258879b798a7a738fbae7e833e1ce723" Oct 29 22:15:04.819388 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:04.819361 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22_odh-ai-gateway-infra(30391eb2-4ecd-4a26-9e63-27d1284a338c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" Oct 29 22:15:04.821143 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.821121 4952 generic.go:356] "Generic (PLEG): container finished" podID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerID="f46ebedb74def8853ee101edefb94bb333be2dbaf29292a9730a89bcfce44da6" exitCode=60 Oct 29 22:15:04.821466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.821443 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" event={"ID":"41f63e45-db88-4f25-aeb3-7f8cb5cec37e","Type":"ContainerDied","Data":"f46ebedb74def8853ee101edefb94bb333be2dbaf29292a9730a89bcfce44da6"} Oct 29 22:15:04.821663 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.821647 4952 scope.go:122] "RemoveContainer" containerID="f46ebedb74def8853ee101edefb94bb333be2dbaf29292a9730a89bcfce44da6" Oct 29 22:15:04.822093 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:04.822071 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2_odh-ai-gateway-infra(41f63e45-db88-4f25-aeb3-7f8cb5cec37e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" Oct 29 22:15:04.823770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.823744 4952 generic.go:356] "Generic (PLEG): container finished" podID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerID="0923260274f691fdc543485a6e4ea3e0b8784d8ad6e4dc8a2c941ef85513330c" exitCode=60 Oct 29 22:15:04.823875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.823830 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" event={"ID":"f7fa4c80-4284-41f6-8dde-d761102e4138","Type":"ContainerDied","Data":"0923260274f691fdc543485a6e4ea3e0b8784d8ad6e4dc8a2c941ef85513330c"} Oct 29 22:15:04.824341 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.824292 4952 scope.go:122] "RemoveContainer" containerID="0923260274f691fdc543485a6e4ea3e0b8784d8ad6e4dc8a2c941ef85513330c" Oct 29 22:15:04.824979 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:04.824639 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv_odh-ai-gateway-infra(f7fa4c80-4284-41f6-8dde-d761102e4138)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" Oct 29 22:15:04.826339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.826318 4952 generic.go:356] "Generic (PLEG): container finished" podID="93ea755a-2d21-44a4-9604-564c464c872f" containerID="4a906722b19a94a89890111a95066a94e218cb3dbbde37aa47447f9b9ea5b23a" exitCode=60 Oct 29 22:15:04.826461 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.826399 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" event={"ID":"93ea755a-2d21-44a4-9604-564c464c872f","Type":"ContainerDied","Data":"4a906722b19a94a89890111a95066a94e218cb3dbbde37aa47447f9b9ea5b23a"} Oct 29 22:15:04.826691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.826666 4952 scope.go:122] "RemoveContainer" containerID="4a906722b19a94a89890111a95066a94e218cb3dbbde37aa47447f9b9ea5b23a" Oct 29 22:15:04.827090 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:04.827064 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888535-2cdmt_odh-ai-gateway-infra(93ea755a-2d21-44a4-9604-564c464c872f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" podUID="93ea755a-2d21-44a4-9604-564c464c872f" Oct 29 22:15:04.828698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.828676 4952 generic.go:356] "Generic (PLEG): container finished" podID="6e683892-5640-48ab-8737-b02f21a939a1" containerID="bbbacf93a141ba4f1cef9093ec6a69f4d5e55186f15680d4cfaca26798ab8625" exitCode=60 Oct 29 22:15:04.828813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.828743 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" event={"ID":"6e683892-5640-48ab-8737-b02f21a939a1","Type":"ContainerDied","Data":"bbbacf93a141ba4f1cef9093ec6a69f4d5e55186f15680d4cfaca26798ab8625"} Oct 29 22:15:04.829089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.829067 4952 scope.go:122] "RemoveContainer" containerID="bbbacf93a141ba4f1cef9093ec6a69f4d5e55186f15680d4cfaca26798ab8625" Oct 29 22:15:04.829345 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:04.829326 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc_odh-ai-gateway-infra(6e683892-5640-48ab-8737-b02f21a939a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" podUID="6e683892-5640-48ab-8737-b02f21a939a1" Oct 29 22:15:04.830852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.830830 4952 scope.go:122] "RemoveContainer" containerID="0fa392b2c2824420610db0cadfef46113aa8865024f5bab9a598bc9f0ee91bfc" Oct 29 22:15:04.841523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.841503 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:15:04.845479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.845464 4952 scope.go:122] "RemoveContainer" containerID="df0a47834ab55ab7fa8c405cde70258faa470df37328a814f09119e213dc3dab" Oct 29 22:15:04.861009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.860984 4952 scope.go:122] "RemoveContainer" containerID="6eb1b16c1bf622187fc663945255a1f98f65b6c2eca0ac17a0eb2e47055ca472" Oct 29 22:15:04.875211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.875194 4952 scope.go:122] "RemoveContainer" containerID="32b2c8a9cde0f7db4eb54cf27be9025623f31d9efbca15e6ada65aa1f5cb7455" Oct 29 22:15:04.889086 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.889061 4952 scope.go:122] "RemoveContainer" containerID="3f4c36fce359f6797a26e4b7ce76cc77c690a666214bf250c9d35c30f94ad2bb" Oct 29 22:15:04.903792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:04.903773 4952 scope.go:122] "RemoveContainer" containerID="202fc05bd7297a5df6b2461a3981ab0a0fc1e880745d3041f7c4e47bc81aa422" Oct 29 22:15:05.845477 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:05.845444 4952 scope.go:122] "RemoveContainer" containerID="b33a3c262fd7ee07a5ae9e49d9857250d19b9aa820ed0295f552d5587f6df0be" Oct 29 22:15:05.846026 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:05.845866 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr_odh-ai-gateway-infra(610a2b6f-c986-4c94-834c-b17f655caba7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" Oct 29 22:15:05.847556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:05.847533 4952 scope.go:122] "RemoveContainer" containerID="bea779102f0ccf43fd962f903f03fd8b258879b798a7a738fbae7e833e1ce723" Oct 29 22:15:05.847942 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:05.847899 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22_odh-ai-gateway-infra(30391eb2-4ecd-4a26-9e63-27d1284a338c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" Oct 29 22:15:05.850245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:05.850224 4952 scope.go:122] "RemoveContainer" containerID="f46ebedb74def8853ee101edefb94bb333be2dbaf29292a9730a89bcfce44da6" Oct 29 22:15:05.850556 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:05.850530 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2_odh-ai-gateway-infra(41f63e45-db88-4f25-aeb3-7f8cb5cec37e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" Oct 29 22:15:05.852375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:05.852352 4952 scope.go:122] "RemoveContainer" containerID="0923260274f691fdc543485a6e4ea3e0b8784d8ad6e4dc8a2c941ef85513330c" Oct 29 22:15:05.852715 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:05.852689 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv_odh-ai-gateway-infra(f7fa4c80-4284-41f6-8dde-d761102e4138)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" Oct 29 22:15:05.854399 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:05.854363 4952 scope.go:122] "RemoveContainer" containerID="4a906722b19a94a89890111a95066a94e218cb3dbbde37aa47447f9b9ea5b23a" Oct 29 22:15:05.854680 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:05.854653 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888535-2cdmt_odh-ai-gateway-infra(93ea755a-2d21-44a4-9604-564c464c872f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" podUID="93ea755a-2d21-44a4-9604-564c464c872f" Oct 29 22:15:05.856826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:05.856800 4952 scope.go:122] "RemoveContainer" containerID="bbbacf93a141ba4f1cef9093ec6a69f4d5e55186f15680d4cfaca26798ab8625" Oct 29 22:15:05.857194 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:05.857158 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc_odh-ai-gateway-infra(6e683892-5640-48ab-8737-b02f21a939a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" podUID="6e683892-5640-48ab-8737-b02f21a939a1" Oct 29 22:15:05.858956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:05.858891 4952 scope.go:122] "RemoveContainer" containerID="0c7241e16cd138471d64150fac5dd77e32f7fd1861918268489d4e4fe07259fe" Oct 29 22:15:05.859253 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:05.859230 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx_odh-ai-gateway-infra(ac1fb38f-4266-4290-ae59-94dc484c750f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" Oct 29 22:15:06.254001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:06.251816 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:08.144928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:08.137646 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:09.359157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:09.359097 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:09.911626 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:09.911466 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:11.966978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:11.966941 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:12.291860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:12.291743 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:14.073421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:14.073387 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:14.737204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:14.737167 4952 scope.go:122] "RemoveContainer" containerID="b33a3c262fd7ee07a5ae9e49d9857250d19b9aa820ed0295f552d5587f6df0be" Oct 29 22:15:14.740883 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:14.740851 4952 scope.go:122] "RemoveContainer" containerID="bea779102f0ccf43fd962f903f03fd8b258879b798a7a738fbae7e833e1ce723" Oct 29 22:15:15.565705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:15.565613 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" podUID="cdaaa603-46cc-43ed-a102-9005c08df63f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.131:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:15:15.580516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:15.580187 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:15:15.602107 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:15.602057 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:15:15.637953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:15.636450 4952 scope.go:122] "RemoveContainer" containerID="bbbacf93a141ba4f1cef9093ec6a69f4d5e55186f15680d4cfaca26798ab8625" Oct 29 22:15:15.637953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:15.637475 4952 scope.go:122] "RemoveContainer" containerID="0923260274f691fdc543485a6e4ea3e0b8784d8ad6e4dc8a2c941ef85513330c" Oct 29 22:15:15.654263 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:15.654230 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:15:15.656166 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:15.655466 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:15:16.167710 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:16.167667 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" event={"ID":"610a2b6f-c986-4c94-834c-b17f655caba7","Type":"ContainerStarted","Data":"f93473316290cf14bd08c62715a885f6e573308054e92a988b52b0f16fef78be"} Oct 29 22:15:16.170650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:16.170617 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" event={"ID":"30391eb2-4ecd-4a26-9e63-27d1284a338c","Type":"ContainerStarted","Data":"5f5f506608218c1c4aa5c5a5f962d950f367b62efeb6c8b12f46b7a4371b6d01"} Oct 29 22:15:16.486955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:16.481386 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:16.682933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:16.672482 4952 scope.go:122] "RemoveContainer" containerID="4a906722b19a94a89890111a95066a94e218cb3dbbde37aa47447f9b9ea5b23a" Oct 29 22:15:16.682933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:16.674015 4952 scope.go:122] "RemoveContainer" containerID="f46ebedb74def8853ee101edefb94bb333be2dbaf29292a9730a89bcfce44da6" Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.190207 4952 generic.go:356] "Generic (PLEG): container finished" podID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerID="bffec84092a316df5c145e50c0c2c81223f15b145d81334cff0f63ae640a4477" exitCode=60 Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.190314 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" event={"ID":"f7fa4c80-4284-41f6-8dde-d761102e4138","Type":"ContainerDied","Data":"bffec84092a316df5c145e50c0c2c81223f15b145d81334cff0f63ae640a4477"} Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.190371 4952 scope.go:122] "RemoveContainer" containerID="0923260274f691fdc543485a6e4ea3e0b8784d8ad6e4dc8a2c941ef85513330c" Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.190940 4952 scope.go:122] "RemoveContainer" containerID="bffec84092a316df5c145e50c0c2c81223f15b145d81334cff0f63ae640a4477" Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.194977 4952 generic.go:356] "Generic (PLEG): container finished" podID="6e683892-5640-48ab-8737-b02f21a939a1" containerID="1cc421e0536f502701ae790ef4219db84f4f6686e0d050f0abc3352975386f37" exitCode=60 Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.195036 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" event={"ID":"6e683892-5640-48ab-8737-b02f21a939a1","Type":"ContainerDied","Data":"1cc421e0536f502701ae790ef4219db84f4f6686e0d050f0abc3352975386f37"} Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.195515 4952 scope.go:122] "RemoveContainer" containerID="1cc421e0536f502701ae790ef4219db84f4f6686e0d050f0abc3352975386f37" Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:17.195960 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc_odh-ai-gateway-infra(6e683892-5640-48ab-8737-b02f21a939a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" podUID="6e683892-5640-48ab-8737-b02f21a939a1" Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:17.197542 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv_odh-ai-gateway-infra(f7fa4c80-4284-41f6-8dde-d761102e4138)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.199791 4952 generic.go:356] "Generic (PLEG): container finished" podID="610a2b6f-c986-4c94-834c-b17f655caba7" containerID="f93473316290cf14bd08c62715a885f6e573308054e92a988b52b0f16fef78be" exitCode=60 Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.199874 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" event={"ID":"610a2b6f-c986-4c94-834c-b17f655caba7","Type":"ContainerDied","Data":"f93473316290cf14bd08c62715a885f6e573308054e92a988b52b0f16fef78be"} Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.200880 4952 scope.go:122] "RemoveContainer" containerID="f93473316290cf14bd08c62715a885f6e573308054e92a988b52b0f16fef78be" Oct 29 22:15:17.202926 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:17.201363 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr_odh-ai-gateway-infra(610a2b6f-c986-4c94-834c-b17f655caba7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" Oct 29 22:15:17.204037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.204011 4952 generic.go:356] "Generic (PLEG): container finished" podID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerID="5f5f506608218c1c4aa5c5a5f962d950f367b62efeb6c8b12f46b7a4371b6d01" exitCode=60 Oct 29 22:15:17.204299 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.204085 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" event={"ID":"30391eb2-4ecd-4a26-9e63-27d1284a338c","Type":"ContainerDied","Data":"5f5f506608218c1c4aa5c5a5f962d950f367b62efeb6c8b12f46b7a4371b6d01"} Oct 29 22:15:17.204855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.204815 4952 scope.go:122] "RemoveContainer" containerID="5f5f506608218c1c4aa5c5a5f962d950f367b62efeb6c8b12f46b7a4371b6d01" Oct 29 22:15:17.205485 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:17.205434 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22_odh-ai-gateway-infra(30391eb2-4ecd-4a26-9e63-27d1284a338c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" Oct 29 22:15:17.216602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.216577 4952 scope.go:122] "RemoveContainer" containerID="bbbacf93a141ba4f1cef9093ec6a69f4d5e55186f15680d4cfaca26798ab8625" Oct 29 22:15:17.245114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.245078 4952 scope.go:122] "RemoveContainer" containerID="b33a3c262fd7ee07a5ae9e49d9857250d19b9aa820ed0295f552d5587f6df0be" Oct 29 22:15:17.292495 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.290369 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr"] Oct 29 22:15:17.303044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.303014 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22"] Oct 29 22:15:17.313700 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.313000 4952 scope.go:122] "RemoveContainer" containerID="bea779102f0ccf43fd962f903f03fd8b258879b798a7a738fbae7e833e1ce723" Oct 29 22:15:17.638557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:17.638505 4952 scope.go:122] "RemoveContainer" containerID="0c7241e16cd138471d64150fac5dd77e32f7fd1861918268489d4e4fe07259fe" Oct 29 22:15:18.220797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.220705 4952 generic.go:356] "Generic (PLEG): container finished" podID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerID="6229cc995e04186f1ff703efd1bea0135328543aa3d74e83f7c01b08f0bde7bf" exitCode=60 Oct 29 22:15:18.220797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.220741 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" event={"ID":"41f63e45-db88-4f25-aeb3-7f8cb5cec37e","Type":"ContainerDied","Data":"6229cc995e04186f1ff703efd1bea0135328543aa3d74e83f7c01b08f0bde7bf"} Oct 29 22:15:18.221375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.220815 4952 scope.go:122] "RemoveContainer" containerID="f46ebedb74def8853ee101edefb94bb333be2dbaf29292a9730a89bcfce44da6" Oct 29 22:15:18.230439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.229848 4952 generic.go:356] "Generic (PLEG): container finished" podID="93ea755a-2d21-44a4-9604-564c464c872f" containerID="ff114f5bf3a3b512c11ba2a70b4427189bbd6492620cec5c5f634f814d242978" exitCode=60 Oct 29 22:15:18.230439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.229943 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" event={"ID":"93ea755a-2d21-44a4-9604-564c464c872f","Type":"ContainerDied","Data":"ff114f5bf3a3b512c11ba2a70b4427189bbd6492620cec5c5f634f814d242978"} Oct 29 22:15:18.237949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.234768 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" event={"ID":"ac1fb38f-4266-4290-ae59-94dc484c750f","Type":"ContainerStarted","Data":"d2aa6846d196660a6ae69fa43b6caa993874d724b5e4b9610585ddd9f19168ed"} Oct 29 22:15:18.237949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.236271 4952 scope.go:122] "RemoveContainer" containerID="ff114f5bf3a3b512c11ba2a70b4427189bbd6492620cec5c5f634f814d242978" Oct 29 22:15:18.237949 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:18.236701 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888535-2cdmt_odh-ai-gateway-infra(93ea755a-2d21-44a4-9604-564c464c872f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" podUID="93ea755a-2d21-44a4-9604-564c464c872f" Oct 29 22:15:18.238374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.237973 4952 scope.go:122] "RemoveContainer" containerID="6229cc995e04186f1ff703efd1bea0135328543aa3d74e83f7c01b08f0bde7bf" Oct 29 22:15:18.238374 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:18.238254 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2_odh-ai-gateway-infra(41f63e45-db88-4f25-aeb3-7f8cb5cec37e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" Oct 29 22:15:18.279392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.277027 4952 scope.go:122] "RemoveContainer" containerID="4a906722b19a94a89890111a95066a94e218cb3dbbde37aa47447f9b9ea5b23a" Oct 29 22:15:18.314329 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:18.303098 4952 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f63e45_db88_4f25_aeb3_7f8cb5cec37e.slice/crio-conmon-6229cc995e04186f1ff703efd1bea0135328543aa3d74e83f7c01b08f0bde7bf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ea755a_2d21_44a4_9604_564c464c872f.slice/crio-conmon-ff114f5bf3a3b512c11ba2a70b4427189bbd6492620cec5c5f634f814d242978.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:15:18.314329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.307731 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv"] Oct 29 22:15:18.418216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.416326 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc"] Oct 29 22:15:18.821935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:18.783807 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:19.281017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.280984 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" Oct 29 22:15:19.291171 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.291143 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" Oct 29 22:15:19.294134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.291708 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" Oct 29 22:15:19.315036 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.315009 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" Oct 29 22:15:19.315294 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.315082 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" Oct 29 22:15:19.316248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.315095 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv" event={"ID":"f7fa4c80-4284-41f6-8dde-d761102e4138","Type":"ContainerDied","Data":"0779cbb150a535f3d2af7bbef0ccd20427001e8b568cd898d59bd6043bcf0194"} Oct 29 22:15:19.316366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.316318 4952 scope.go:122] "RemoveContainer" containerID="bffec84092a316df5c145e50c0c2c81223f15b145d81334cff0f63ae640a4477" Oct 29 22:15:19.347931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.344254 4952 generic.go:356] "Generic (PLEG): container finished" podID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerID="d2aa6846d196660a6ae69fa43b6caa993874d724b5e4b9610585ddd9f19168ed" exitCode=60 Oct 29 22:15:19.347931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.344449 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" event={"ID":"ac1fb38f-4266-4290-ae59-94dc484c750f","Type":"ContainerDied","Data":"d2aa6846d196660a6ae69fa43b6caa993874d724b5e4b9610585ddd9f19168ed"} Oct 29 22:15:19.347931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.345015 4952 scope.go:122] "RemoveContainer" containerID="d2aa6846d196660a6ae69fa43b6caa993874d724b5e4b9610585ddd9f19168ed" Oct 29 22:15:19.347931 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:19.345478 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx_odh-ai-gateway-infra(ac1fb38f-4266-4290-ae59-94dc484c750f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" Oct 29 22:15:19.399472 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.360038 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" event={"ID":"610a2b6f-c986-4c94-834c-b17f655caba7","Type":"ContainerDied","Data":"d5aacc15aadfa7b9a7ee8c512a837c3dcf47c74d35da02322b46fbbf0fc32152"} Oct 29 22:15:19.399472 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.360203 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr" Oct 29 22:15:19.399472 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.386830 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" event={"ID":"30391eb2-4ecd-4a26-9e63-27d1284a338c","Type":"ContainerDied","Data":"07862d1e59fd66de435725dc3a8858cdf9f90f37ed6fa60a34f3bc53c6c673a7"} Oct 29 22:15:19.399472 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.386999 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22" Oct 29 22:15:19.399472 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.391568 4952 scope.go:122] "RemoveContainer" containerID="0c7241e16cd138471d64150fac5dd77e32f7fd1861918268489d4e4fe07259fe" Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.418461 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fa4c80-4284-41f6-8dde-d761102e4138-kube-api-access-hgnpp\" (UniqueName: \"kubernetes.io/projected/f7fa4c80-4284-41f6-8dde-d761102e4138-kube-api-access-hgnpp\") pod \"f7fa4c80-4284-41f6-8dde-d761102e4138\" (UID: \"f7fa4c80-4284-41f6-8dde-d761102e4138\") " Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.418829 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e683892-5640-48ab-8737-b02f21a939a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e683892-5640-48ab-8737-b02f21a939a1-maas-api-service-ca\") pod \"6e683892-5640-48ab-8737-b02f21a939a1\" (UID: \"6e683892-5640-48ab-8737-b02f21a939a1\") " Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.419271 4952 scope.go:122] "RemoveContainer" containerID="f93473316290cf14bd08c62715a885f6e573308054e92a988b52b0f16fef78be" Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.434093 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e683892-5640-48ab-8737-b02f21a939a1-maas-api-service-ca" pod "6e683892-5640-48ab-8737-b02f21a939a1" (UID: "6e683892-5640-48ab-8737-b02f21a939a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.439458 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30391eb2-4ecd-4a26-9e63-27d1284a338c-kube-api-access-7gt7b\" (UniqueName: \"kubernetes.io/projected/30391eb2-4ecd-4a26-9e63-27d1284a338c-kube-api-access-7gt7b\") pod \"30391eb2-4ecd-4a26-9e63-27d1284a338c\" (UID: \"30391eb2-4ecd-4a26-9e63-27d1284a338c\") " Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.439635 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fa4c80-4284-41f6-8dde-d761102e4138-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa4c80-4284-41f6-8dde-d761102e4138-maas-api-service-ca\") pod \"f7fa4c80-4284-41f6-8dde-d761102e4138\" (UID: \"f7fa4c80-4284-41f6-8dde-d761102e4138\") " Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.439741 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30391eb2-4ecd-4a26-9e63-27d1284a338c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30391eb2-4ecd-4a26-9e63-27d1284a338c-maas-api-service-ca\") pod \"30391eb2-4ecd-4a26-9e63-27d1284a338c\" (UID: \"30391eb2-4ecd-4a26-9e63-27d1284a338c\") " Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.439975 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e683892-5640-48ab-8737-b02f21a939a1-kube-api-access-b2dcl\" (UniqueName: \"kubernetes.io/projected/6e683892-5640-48ab-8737-b02f21a939a1-kube-api-access-b2dcl\") pod \"6e683892-5640-48ab-8737-b02f21a939a1\" (UID: \"6e683892-5640-48ab-8737-b02f21a939a1\") " Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.440048 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610a2b6f-c986-4c94-834c-b17f655caba7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610a2b6f-c986-4c94-834c-b17f655caba7-maas-api-service-ca\") pod \"610a2b6f-c986-4c94-834c-b17f655caba7\" (UID: \"610a2b6f-c986-4c94-834c-b17f655caba7\") " Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.440170 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610a2b6f-c986-4c94-834c-b17f655caba7-kube-api-access-skkmt\" (UniqueName: \"kubernetes.io/projected/610a2b6f-c986-4c94-834c-b17f655caba7-kube-api-access-skkmt\") pod \"610a2b6f-c986-4c94-834c-b17f655caba7\" (UID: \"610a2b6f-c986-4c94-834c-b17f655caba7\") " Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.441475 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e683892-5640-48ab-8737-b02f21a939a1-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.442449 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fa4c80-4284-41f6-8dde-d761102e4138-maas-api-service-ca" pod "f7fa4c80-4284-41f6-8dde-d761102e4138" (UID: "f7fa4c80-4284-41f6-8dde-d761102e4138"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.442896 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30391eb2-4ecd-4a26-9e63-27d1284a338c-maas-api-service-ca" pod "30391eb2-4ecd-4a26-9e63-27d1284a338c" (UID: "30391eb2-4ecd-4a26-9e63-27d1284a338c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.444349 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610a2b6f-c986-4c94-834c-b17f655caba7-maas-api-service-ca" pod "610a2b6f-c986-4c94-834c-b17f655caba7" (UID: "610a2b6f-c986-4c94-834c-b17f655caba7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.460479 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e683892-5640-48ab-8737-b02f21a939a1-kube-api-access-b2dcl" pod "6e683892-5640-48ab-8737-b02f21a939a1" (UID: "6e683892-5640-48ab-8737-b02f21a939a1"). InnerVolumeSpecName "kube-api-access-b2dcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.464406 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fa4c80-4284-41f6-8dde-d761102e4138-kube-api-access-hgnpp" pod "f7fa4c80-4284-41f6-8dde-d761102e4138" (UID: "f7fa4c80-4284-41f6-8dde-d761102e4138"). InnerVolumeSpecName "kube-api-access-hgnpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:19.485085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.478558 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610a2b6f-c986-4c94-834c-b17f655caba7-kube-api-access-skkmt" pod "610a2b6f-c986-4c94-834c-b17f655caba7" (UID: "610a2b6f-c986-4c94-834c-b17f655caba7"). InnerVolumeSpecName "kube-api-access-skkmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:19.505968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.504779 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30391eb2-4ecd-4a26-9e63-27d1284a338c-kube-api-access-7gt7b" pod "30391eb2-4ecd-4a26-9e63-27d1284a338c" (UID: "30391eb2-4ecd-4a26-9e63-27d1284a338c"). InnerVolumeSpecName "kube-api-access-7gt7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:19.505968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.505372 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2"] Oct 29 22:15:19.592934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.522013 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt"] Oct 29 22:15:19.593449 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.543830 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgnpp\" (UniqueName: \"kubernetes.io/projected/f7fa4c80-4284-41f6-8dde-d761102e4138-kube-api-access-hgnpp\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.593576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.593475 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gt7b\" (UniqueName: \"kubernetes.io/projected/30391eb2-4ecd-4a26-9e63-27d1284a338c-kube-api-access-7gt7b\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.593576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.593495 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa4c80-4284-41f6-8dde-d761102e4138-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.593576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.593513 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30391eb2-4ecd-4a26-9e63-27d1284a338c-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.593576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.593529 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2dcl\" (UniqueName: \"kubernetes.io/projected/6e683892-5640-48ab-8737-b02f21a939a1-kube-api-access-b2dcl\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.593576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.593544 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610a2b6f-c986-4c94-834c-b17f655caba7-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.593576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.593559 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skkmt\" (UniqueName: \"kubernetes.io/projected/610a2b6f-c986-4c94-834c-b17f655caba7-kube-api-access-skkmt\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:19.600607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.600582 4952 scope.go:122] "RemoveContainer" containerID="5f5f506608218c1c4aa5c5a5f962d950f367b62efeb6c8b12f46b7a4371b6d01" Oct 29 22:15:19.611106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.611070 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx"] Oct 29 22:15:19.844480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.844409 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv"] Oct 29 22:15:19.906172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.902886 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-971004-29888535-r6gcv"] Oct 29 22:15:19.983934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.980973 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22"] Oct 29 22:15:19.990963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:19.990768 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888535-xwx22"] Oct 29 22:15:20.002705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.002031 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" Oct 29 22:15:20.073926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.073152 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr"] Oct 29 22:15:20.077947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.077166 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888535-lmcpr"] Oct 29 22:15:20.106509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.106466 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-maas-api-service-ca\") pod \"41f63e45-db88-4f25-aeb3-7f8cb5cec37e\" (UID: \"41f63e45-db88-4f25-aeb3-7f8cb5cec37e\") " Oct 29 22:15:20.106682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.106667 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-kube-api-access-vbsdd\" (UniqueName: \"kubernetes.io/projected/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-kube-api-access-vbsdd\") pod \"41f63e45-db88-4f25-aeb3-7f8cb5cec37e\" (UID: \"41f63e45-db88-4f25-aeb3-7f8cb5cec37e\") " Oct 29 22:15:20.106892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.106866 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-maas-api-service-ca" pod "41f63e45-db88-4f25-aeb3-7f8cb5cec37e" (UID: "41f63e45-db88-4f25-aeb3-7f8cb5cec37e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:20.107568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.107541 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:20.109233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.109208 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-kube-api-access-vbsdd" pod "41f63e45-db88-4f25-aeb3-7f8cb5cec37e" (UID: "41f63e45-db88-4f25-aeb3-7f8cb5cec37e"). InnerVolumeSpecName "kube-api-access-vbsdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:20.209761 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.209212 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbsdd\" (UniqueName: \"kubernetes.io/projected/41f63e45-db88-4f25-aeb3-7f8cb5cec37e-kube-api-access-vbsdd\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:20.271952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.271870 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" Oct 29 22:15:20.311416 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.311386 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ea755a-2d21-44a4-9604-564c464c872f-kube-api-access-znmmr\" (UniqueName: \"kubernetes.io/projected/93ea755a-2d21-44a4-9604-564c464c872f-kube-api-access-znmmr\") pod \"93ea755a-2d21-44a4-9604-564c464c872f\" (UID: \"93ea755a-2d21-44a4-9604-564c464c872f\") " Oct 29 22:15:20.311825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.311619 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ea755a-2d21-44a4-9604-564c464c872f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ea755a-2d21-44a4-9604-564c464c872f-maas-api-service-ca\") pod \"93ea755a-2d21-44a4-9604-564c464c872f\" (UID: \"93ea755a-2d21-44a4-9604-564c464c872f\") " Oct 29 22:15:20.312591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.312556 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ea755a-2d21-44a4-9604-564c464c872f-maas-api-service-ca" pod "93ea755a-2d21-44a4-9604-564c464c872f" (UID: "93ea755a-2d21-44a4-9604-564c464c872f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:20.315967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.314869 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ea755a-2d21-44a4-9604-564c464c872f-kube-api-access-znmmr" pod "93ea755a-2d21-44a4-9604-564c464c872f" (UID: "93ea755a-2d21-44a4-9604-564c464c872f"). InnerVolumeSpecName "kube-api-access-znmmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:20.447309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.414501 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ea755a-2d21-44a4-9604-564c464c872f-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:20.447309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.414541 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znmmr\" (UniqueName: \"kubernetes.io/projected/93ea755a-2d21-44a4-9604-564c464c872f-kube-api-access-znmmr\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:20.723925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.717702 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" path="/var/lib/kubelet/pods/30391eb2-4ecd-4a26-9e63-27d1284a338c/volumes" Oct 29 22:15:20.732555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.726547 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" path="/var/lib/kubelet/pods/610a2b6f-c986-4c94-834c-b17f655caba7/volumes" Oct 29 22:15:20.732555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.727194 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" path="/var/lib/kubelet/pods/f7fa4c80-4284-41f6-8dde-d761102e4138/volumes" Oct 29 22:15:20.804089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.802612 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" event={"ID":"41f63e45-db88-4f25-aeb3-7f8cb5cec37e","Type":"ContainerDied","Data":"28d38e65983fa4d0b03f4e8aa527badac0cd95fd89d5bd0d11d05f481a95a7af"} Oct 29 22:15:20.804089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.802688 4952 scope.go:122] "RemoveContainer" containerID="6229cc995e04186f1ff703efd1bea0135328543aa3d74e83f7c01b08f0bde7bf" Oct 29 22:15:20.804089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.802790 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2" Oct 29 22:15:20.810207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.810178 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" Oct 29 22:15:20.810329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.810176 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt" event={"ID":"93ea755a-2d21-44a4-9604-564c464c872f","Type":"ContainerDied","Data":"dbe850b14279d1ada4f83f4f4b90c8a68567f62ee72c493aa0d7333bf1a7c61e"} Oct 29 22:15:20.811536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.811512 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" Oct 29 22:15:20.811622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.811543 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc" event={"ID":"6e683892-5640-48ab-8737-b02f21a939a1","Type":"ContainerDied","Data":"dfe2f5791d242c01e765bffda07dfde714fe666d322722c689b6fff23211c4a5"} Oct 29 22:15:20.863622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.863586 4952 scope.go:122] "RemoveContainer" containerID="ff114f5bf3a3b512c11ba2a70b4427189bbd6492620cec5c5f634f814d242978" Oct 29 22:15:20.878871 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.878849 4952 scope.go:122] "RemoveContainer" containerID="1cc421e0536f502701ae790ef4219db84f4f6686e0d050f0abc3352975386f37" Oct 29 22:15:20.996708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:20.996499 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2"] Oct 29 22:15:21.048930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.030248 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888535-cthv2"] Oct 29 22:15:21.053479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.053442 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:21.135486 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.135178 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc"] Oct 29 22:15:21.142632 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.142596 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888535-4whxc"] Oct 29 22:15:21.152515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.152484 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt"] Oct 29 22:15:21.161575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.161543 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-2cdmt"] Oct 29 22:15:21.254491 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.254410 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" Oct 29 22:15:21.288773 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.288741 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1fb38f-4266-4290-ae59-94dc484c750f-kube-api-access-kmrl8\" (UniqueName: \"kubernetes.io/projected/ac1fb38f-4266-4290-ae59-94dc484c750f-kube-api-access-kmrl8\") pod \"ac1fb38f-4266-4290-ae59-94dc484c750f\" (UID: \"ac1fb38f-4266-4290-ae59-94dc484c750f\") " Oct 29 22:15:21.288984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.288804 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1fb38f-4266-4290-ae59-94dc484c750f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1fb38f-4266-4290-ae59-94dc484c750f-maas-api-service-ca\") pod \"ac1fb38f-4266-4290-ae59-94dc484c750f\" (UID: \"ac1fb38f-4266-4290-ae59-94dc484c750f\") " Oct 29 22:15:21.289880 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.289848 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1fb38f-4266-4290-ae59-94dc484c750f-maas-api-service-ca" pod "ac1fb38f-4266-4290-ae59-94dc484c750f" (UID: "ac1fb38f-4266-4290-ae59-94dc484c750f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:15:21.292165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.292108 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1fb38f-4266-4290-ae59-94dc484c750f-kube-api-access-kmrl8" pod "ac1fb38f-4266-4290-ae59-94dc484c750f" (UID: "ac1fb38f-4266-4290-ae59-94dc484c750f"). InnerVolumeSpecName "kube-api-access-kmrl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:21.391604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.391565 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmrl8\" (UniqueName: \"kubernetes.io/projected/ac1fb38f-4266-4290-ae59-94dc484c750f-kube-api-access-kmrl8\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:21.391604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.391602 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1fb38f-4266-4290-ae59-94dc484c750f-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:21.631961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.631092 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:21.847034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.843509 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" event={"ID":"ac1fb38f-4266-4290-ae59-94dc484c750f","Type":"ContainerDied","Data":"9576b39bd9a36c40e4ea3822e1a8d7df19027b4150b5010dc0e60942e5918055"} Oct 29 22:15:21.847034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.843584 4952 scope.go:122] "RemoveContainer" containerID="d2aa6846d196660a6ae69fa43b6caa993874d724b5e4b9610585ddd9f19168ed" Oct 29 22:15:21.847034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.843738 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx" Oct 29 22:15:21.911075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.910534 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx"] Oct 29 22:15:21.923656 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:21.923616 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-863f1b-29888535-gpnhx"] Oct 29 22:15:22.074806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:22.074771 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:22.586932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:22.583560 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:22.849980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:22.791496 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" path="/var/lib/kubelet/pods/41f63e45-db88-4f25-aeb3-7f8cb5cec37e/volumes" Oct 29 22:15:22.851456 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:22.851423 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e683892-5640-48ab-8737-b02f21a939a1" path="/var/lib/kubelet/pods/6e683892-5640-48ab-8737-b02f21a939a1/volumes" Oct 29 22:15:22.852359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:22.852337 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ea755a-2d21-44a4-9604-564c464c872f" path="/var/lib/kubelet/pods/93ea755a-2d21-44a4-9604-564c464c872f/volumes" Oct 29 22:15:22.863965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:22.842249 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:22.864285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:22.864262 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" path="/var/lib/kubelet/pods/ac1fb38f-4266-4290-ae59-94dc484c750f/volumes" Oct 29 22:15:23.145985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:23.145607 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:23.418545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:23.418460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:25.149809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:25.148369 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:26.412587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:26.412493 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:26.577200 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:26.577166 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:26.982985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:26.981920 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:27.006357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:27.006317 4952 generic.go:356] "Generic (PLEG): container finished" podID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerID="c36746517a2d58b39e1afad171c5503974227c8c7aefe38d004de031be7e156c" exitCode=0 Oct 29 22:15:27.006558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:27.006497 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" event={"ID":"345f5596-f75f-4aa1-adc4-d97f16ffa1a0","Type":"ContainerDied","Data":"c36746517a2d58b39e1afad171c5503974227c8c7aefe38d004de031be7e156c"} Oct 29 22:15:27.007085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:27.007062 4952 scope.go:122] "RemoveContainer" containerID="c36746517a2d58b39e1afad171c5503974227c8c7aefe38d004de031be7e156c" Oct 29 22:15:27.011141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:27.011094 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:15:27.011267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:27.011199 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:15:27.011267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:27.011246 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:15:28.070090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:28.070049 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" event={"ID":"345f5596-f75f-4aa1-adc4-d97f16ffa1a0","Type":"ContainerStarted","Data":"e92f37f886a8623f2a4151fc151cecb529978a1cf773b2f762481247ee783e43"} Oct 29 22:15:28.165804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:28.165764 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:28.600812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:28.540554 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:28.990180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:28.990047 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:15:28.990392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:28.990198 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:15:28.990392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:28.990256 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 29 22:15:29.465507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:29.465421 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:29.499103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:29.499006 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:15:29.963465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:29.963414 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:15:29.963669 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:29.963516 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 29 22:15:30.963568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:30.963528 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:15:30.964017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:30.963612 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 29 22:15:31.413928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:31.408788 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:31.963629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:31.963583 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:15:31.964171 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:31.963651 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 29 22:15:32.963410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:32.963365 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 29 22:15:32.963630 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:32.963456 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 29 22:15:33.283398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:33.283234 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:33.468489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:33.468452 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:34.188505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:34.188469 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:15:34.189709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:34.189682 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:15:35.022932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:35.020514 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:36.783075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:36.782968 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:38.463746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:38.463690 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:39.293372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:39.293331 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs"] Oct 29 22:15:39.365920 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:39.365864 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:40.394571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:40.394526 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:42.057357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.057318 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:42.365601 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.365562 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:42.529506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.529469 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:42.843926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.843871 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97"] Oct 29 22:15:42.845050 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845017 4952 cpu_manager.go:414] "containerMap: removing container" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" containerName="cleanup" Oct 29 22:15:42.845195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845059 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" containerName="cleanup" Oct 29 22:15:42.845195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845102 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6e683892-5640-48ab-8737-b02f21a939a1" containerName="cleanup" Oct 29 22:15:42.845195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845111 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e683892-5640-48ab-8737-b02f21a939a1" containerName="cleanup" Oct 29 22:15:42.845195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845123 4952 cpu_manager.go:414] "containerMap: removing container" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" containerName="cleanup" Oct 29 22:15:42.845195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845131 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" containerName="cleanup" Oct 29 22:15:42.845195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845148 4952 cpu_manager.go:414] "containerMap: removing container" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerName="cleanup" Oct 29 22:15:42.845195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845156 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerName="cleanup" Oct 29 22:15:42.845195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845171 4952 cpu_manager.go:414] "containerMap: removing container" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerName="cleanup" Oct 29 22:15:42.845195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845178 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerName="cleanup" Oct 29 22:15:42.845195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845195 4952 cpu_manager.go:414] "containerMap: removing container" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845204 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845217 4952 cpu_manager.go:414] "containerMap: removing container" podUID="93ea755a-2d21-44a4-9604-564c464c872f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845225 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ea755a-2d21-44a4-9604-564c464c872f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845238 4952 cpu_manager.go:414] "containerMap: removing container" podUID="93ea755a-2d21-44a4-9604-564c464c872f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845246 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ea755a-2d21-44a4-9604-564c464c872f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845257 4952 cpu_manager.go:414] "containerMap: removing container" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845263 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845280 4952 cpu_manager.go:414] "containerMap: removing container" podUID="93ea755a-2d21-44a4-9604-564c464c872f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845287 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ea755a-2d21-44a4-9604-564c464c872f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845302 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6e683892-5640-48ab-8737-b02f21a939a1" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845309 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e683892-5640-48ab-8737-b02f21a939a1" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845324 4952 cpu_manager.go:414] "containerMap: removing container" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845331 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845340 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845347 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845365 4952 cpu_manager.go:414] "containerMap: removing container" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845372 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845389 4952 cpu_manager.go:414] "containerMap: removing container" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845396 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845414 4952 cpu_manager.go:414] "containerMap: removing container" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845421 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845447 4952 cpu_manager.go:414] "containerMap: removing container" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845454 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845463 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845470 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845486 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845493 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerName="cleanup" Oct 29 22:15:42.845679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845683 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845699 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845709 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e683892-5640-48ab-8737-b02f21a939a1" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845718 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845729 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845741 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e683892-5640-48ab-8737-b02f21a939a1" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845752 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845762 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845772 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845782 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f63e45-db88-4f25-aeb3-7f8cb5cec37e" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845790 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ea755a-2d21-44a4-9604-564c464c872f" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845800 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ea755a-2d21-44a4-9604-564c464c872f" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845809 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845820 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ea755a-2d21-44a4-9604-564c464c872f" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845831 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e683892-5640-48ab-8737-b02f21a939a1" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845841 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845849 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1fb38f-4266-4290-ae59-94dc484c750f" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845858 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerName="cleanup" Oct 29 22:15:42.847225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.845868 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fa4c80-4284-41f6-8dde-d761102e4138" containerName="cleanup" Oct 29 22:15:42.849419 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.849395 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.867554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.867521 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istiod-ca-cert\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.867891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.867745 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-podinfo\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.868168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.868036 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnln\" (UniqueName: \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-kube-api-access-jcnln\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.868623 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.868482 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-data\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.868868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.868849 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-token\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.869112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.869091 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-certs\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.869315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.869297 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.869574 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.869555 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-credential-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.869747 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.869731 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-envoy\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.884104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.884069 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-76c1b7-openshift-default-dockercfg-rvvsj\"" Oct 29 22:15:42.959481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.958977 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97"] Oct 29 22:15:42.972470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.972419 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-certs\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.972628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.972548 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.972628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.972596 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-credential-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.972704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.972671 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-envoy\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.972846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.972830 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istiod-ca-cert\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.972892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.972879 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-podinfo\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.973003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.972922 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnln\" (UniqueName: \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-kube-api-access-jcnln\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.973003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.972960 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-data\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.973076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.973001 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-token\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.973945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.973921 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-certs\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.975202 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.975176 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.975464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.975442 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-credential-socket\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.975716 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.975693 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-data\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.976145 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.976125 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istiod-ca-cert\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.979499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.979472 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-podinfo\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:42.979771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:42.979725 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-envoy\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:43.088928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:43.085770 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:43.186539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:43.184805 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnln\" (UniqueName: \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-kube-api-access-jcnln\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:43.231972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:43.230690 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-token\") pod \"e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:43.266943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:43.261436 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:43.496654 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:43.496403 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:43.890241 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:43.890189 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97"] Oct 29 22:15:43.896858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:43.896818 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:15:43.897096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:43.897072 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:15:43.897257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:43.897240 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:15:44.322980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:44.321553 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" event={"ID":"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b","Type":"ContainerStarted","Data":"7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586"} Oct 29 22:15:44.322980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:44.321618 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" event={"ID":"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b","Type":"ContainerStarted","Data":"7460f53e2aa31f4cfc86e4cef56c068d2dccdaefa4e3efddf8516c5c964f8e54"} Oct 29 22:15:44.326928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:44.324438 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:44.503925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:44.502091 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:44.504152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:44.504036 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:15:44.504152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:44.504098 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 22:15:45.497464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:45.497428 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:15:45.498117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:45.498082 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 22:15:45.563203 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:45.563141 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:15:46.271231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:46.271180 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:46.497758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:46.497714 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:15:46.498237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:46.497784 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 22:15:46.766516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:46.766467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:46.944927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:46.943875 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:47.497620 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:47.497583 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:15:47.497785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:47.497639 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 22:15:48.300007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:48.295996 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:48.498157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:48.498114 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:15:48.498595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:48.498168 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 22:15:48.570639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:48.570403 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:48.901017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:48.896224 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:49.498623 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:49.498589 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:15:49.499060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:49.498641 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 22:15:50.309967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:50.309896 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:50.498311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:50.498264 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:15:50.498534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:50.498344 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 29 22:15:50.514535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:50.514492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:51.160465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:51.160429 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:51.335535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:51.335497 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:51.502708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:51.502633 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:51.502955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:51.502933 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:51.503783 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:51.503763 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:15:51.565390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:51.565338 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" podStartSLOduration=9.565325212 podStartE2EDuration="9.565325212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:44.376857575 +0000 UTC m=+2009.851445998" watchObservedRunningTime="2026-10-29 22:15:51.565325212 +0000 UTC m=+2017.039913629" Oct 29 22:15:52.119082 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:52.119044 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:53.934486 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.934449 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz"] Oct 29 22:15:53.935484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.935458 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6e683892-5640-48ab-8737-b02f21a939a1" containerName="cleanup" Oct 29 22:15:53.935484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.935484 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e683892-5640-48ab-8737-b02f21a939a1" containerName="cleanup" Oct 29 22:15:53.935690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.935565 4952 cpu_manager.go:414] "containerMap: removing container" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" containerName="cleanup" Oct 29 22:15:53.935690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.935573 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" containerName="cleanup" Oct 29 22:15:53.935811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.935756 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="30391eb2-4ecd-4a26-9e63-27d1284a338c" containerName="cleanup" Oct 29 22:15:53.935811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.935788 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="610a2b6f-c986-4c94-834c-b17f655caba7" containerName="cleanup" Oct 29 22:15:53.939489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.939464 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:53.960829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.960801 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz"] Oct 29 22:15:53.966715 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.966690 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-1e74c3\"/\"kube-root-ca.crt\"" Oct 29 22:15:53.967063 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.966690 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-1e74c3\"/\"openshift-service-ca.crt\"" Oct 29 22:15:53.967063 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.966740 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-1e74c3\"/\"default-dockercfg-4vdjs\"" Oct 29 22:15:53.967063 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:53.966774 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-1e74c3\"/\"auth-test-model-1e74c3-kserve-self-signed-certs\"" Oct 29 22:15:54.030866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.030830 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:54.088698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.088656 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-tmp-dir\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.088989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.088966 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-model-cache\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.089146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.089129 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msll\" (UniqueName: \"kubernetes.io/projected/465af292-e674-41e0-9047-733f7fdf93ce-kube-api-access-9msll\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.089215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.089169 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465af292-e674-41e0-9047-733f7fdf93ce-tls-certs\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.089303 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.089282 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-dshm\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.089411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.089389 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-home\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.214983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.198183 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msll\" (UniqueName: \"kubernetes.io/projected/465af292-e674-41e0-9047-733f7fdf93ce-kube-api-access-9msll\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.214983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.198247 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465af292-e674-41e0-9047-733f7fdf93ce-tls-certs\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.214983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.198308 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-dshm\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.214983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.198381 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-home\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.214983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.198453 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-tmp-dir\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.214983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.198646 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-model-cache\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.214983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.199243 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-model-cache\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.214983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.208515 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-home\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.218927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.217289 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-dshm\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.218927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.218735 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:54.229962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.224709 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-tmp-dir\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.246038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.241571 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465af292-e674-41e0-9047-733f7fdf93ce-tls-certs\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.246944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.246861 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msll\" (UniqueName: \"kubernetes.io/projected/465af292-e674-41e0-9047-733f7fdf93ce-kube-api-access-9msll\") pod \"auth-test-model-1e74c3-kserve-855fddffbf-kzgvz\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.268044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.267210 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:54.351926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.351211 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" podUID="9825f391-5b6d-4575-820f-62febeeb352b" containerName="main" containerID="cri-o://0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d" gracePeriod=45 Oct 29 22:15:54.559020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.550867 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" podUID="9825f391-5b6d-4575-820f-62febeeb352b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.137:8000/ready\": dial tcp 10.132.0.137:8000: connect: connection refused" Oct 29 22:15:54.723348 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.723316 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:15:54.764615 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:15:54.764583 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod465af292_e674_41e0_9047_733f7fdf93ce.slice/crio-e8ed837e527f62b132d66a7cc34c00e9cea5ba0ed93ba95d586bdacdc052e1b1 WatchSource:0}: Error finding container e8ed837e527f62b132d66a7cc34c00e9cea5ba0ed93ba95d586bdacdc052e1b1: Status 404 returned error can't find the container with id e8ed837e527f62b132d66a7cc34c00e9cea5ba0ed93ba95d586bdacdc052e1b1 Oct 29 22:15:54.764914 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.764878 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz"] Oct 29 22:15:54.814733 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.814700 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9825f391-5b6d-4575-820f-62febeeb352b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9825f391-5b6d-4575-820f-62febeeb352b-tls-certs\") pod \"9825f391-5b6d-4575-820f-62febeeb352b\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " Oct 29 22:15:54.815007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.814983 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-model-cache\") pod \"9825f391-5b6d-4575-820f-62febeeb352b\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " Oct 29 22:15:54.815207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.815178 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-dshm\") pod \"9825f391-5b6d-4575-820f-62febeeb352b\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " Oct 29 22:15:54.815311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.815227 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-home\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-home\") pod \"9825f391-5b6d-4575-820f-62febeeb352b\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " Oct 29 22:15:54.815398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.815334 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-model-cache" pod "9825f391-5b6d-4575-820f-62febeeb352b" (UID: "9825f391-5b6d-4575-820f-62febeeb352b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:15:54.815398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.815367 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9825f391-5b6d-4575-820f-62febeeb352b-kube-api-access-4xl89\" (UniqueName: \"kubernetes.io/projected/9825f391-5b6d-4575-820f-62febeeb352b-kube-api-access-4xl89\") pod \"9825f391-5b6d-4575-820f-62febeeb352b\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " Oct 29 22:15:54.815550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.815486 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-tmp-dir\") pod \"9825f391-5b6d-4575-820f-62febeeb352b\" (UID: \"9825f391-5b6d-4575-820f-62febeeb352b\") " Oct 29 22:15:54.815882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.815851 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-home" pod "9825f391-5b6d-4575-820f-62febeeb352b" (UID: "9825f391-5b6d-4575-820f-62febeeb352b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:15:54.816100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.816081 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-tmp-dir" pod "9825f391-5b6d-4575-820f-62febeeb352b" (UID: "9825f391-5b6d-4575-820f-62febeeb352b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:15:54.816253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.816228 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:54.816327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.816258 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:54.816327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.816271 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:54.818691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.818672 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9825f391-5b6d-4575-820f-62febeeb352b-tls-certs" pod "9825f391-5b6d-4575-820f-62febeeb352b" (UID: "9825f391-5b6d-4575-820f-62febeeb352b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:15:54.818797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.818776 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-dshm" pod "9825f391-5b6d-4575-820f-62febeeb352b" (UID: "9825f391-5b6d-4575-820f-62febeeb352b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:15:54.819314 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.819291 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9825f391-5b6d-4575-820f-62febeeb352b-kube-api-access-4xl89" pod "9825f391-5b6d-4575-820f-62febeeb352b" (UID: "9825f391-5b6d-4575-820f-62febeeb352b"). InnerVolumeSpecName "kube-api-access-4xl89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:15:54.918336 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.918303 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9825f391-5b6d-4575-820f-62febeeb352b-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:54.918336 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.918331 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9825f391-5b6d-4575-820f-62febeeb352b-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:54.918336 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:54.918340 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xl89\" (UniqueName: \"kubernetes.io/projected/9825f391-5b6d-4575-820f-62febeeb352b-kube-api-access-4xl89\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:15:55.511115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.511017 4952 generic.go:356] "Generic (PLEG): container finished" podID="9825f391-5b6d-4575-820f-62febeeb352b" containerID="0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d" exitCode=0 Oct 29 22:15:55.511115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.511088 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" Oct 29 22:15:55.511628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.511100 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" event={"ID":"9825f391-5b6d-4575-820f-62febeeb352b","Type":"ContainerDied","Data":"0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d"} Oct 29 22:15:55.511628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.511156 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs" event={"ID":"9825f391-5b6d-4575-820f-62febeeb352b","Type":"ContainerDied","Data":"5bdec6dabcbe186de165d3797410db2f1f70a4233dfdf4f8805ff995b2751b45"} Oct 29 22:15:55.511628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.511176 4952 scope.go:122] "RemoveContainer" containerID="0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d" Oct 29 22:15:55.514078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.514047 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" event={"ID":"465af292-e674-41e0-9047-733f7fdf93ce","Type":"ContainerStarted","Data":"9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc"} Oct 29 22:15:55.514218 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.514092 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" event={"ID":"465af292-e674-41e0-9047-733f7fdf93ce","Type":"ContainerStarted","Data":"e8ed837e527f62b132d66a7cc34c00e9cea5ba0ed93ba95d586bdacdc052e1b1"} Oct 29 22:15:55.527230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.527201 4952 scope.go:122] "RemoveContainer" containerID="0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d" Oct 29 22:15:55.528884 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:15:55.528849 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d\": container with ID starting with 0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d not found: ID does not exist" containerID="0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d" Oct 29 22:15:55.529059 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.528897 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d"} err="failed to get container status \"0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d\": rpc error: code = NotFound desc = could not find container \"0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d\": container with ID starting with 0b57787a4f51b05970bcd57b18c6a9104f136226ec9db89ca0e027a32e71075d not found: ID does not exist" Oct 29 22:15:55.574018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.573953 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" podStartSLOduration=2.573932487 podStartE2EDuration="2.573932487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:55.568164263 +0000 UTC m=+2021.042752681" watchObservedRunningTime="2026-10-29 22:15:55.573932487 +0000 UTC m=+2021.048520906" Oct 29 22:15:55.608512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.608476 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs"] Oct 29 22:15:55.615025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.614990 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-971004/autores-model-971004-kserve-65c6bc8dfc-wpxrs"] Oct 29 22:15:55.949125 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:55.948636 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:56.268424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:56.268327 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:15:56.651926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:56.651114 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9825f391-5b6d-4575-820f-62febeeb352b" path="/var/lib/kubelet/pods/9825f391-5b6d-4575-820f-62febeeb352b/volumes" Oct 29 22:15:57.684078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:57.684027 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:59.039940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:59.039868 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2"] Oct 29 22:15:59.044568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:59.040163 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" podUID="6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca" containerName="maas-api" containerID="cri-o://617866489a9ec9463a1170119181595771f4b1851be5c1778a5001066b1ae289" gracePeriod=30 Oct 29 22:15:59.159872 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:59.159839 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:59.847867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:59.847580 4952 generic.go:356] "Generic (PLEG): container finished" podID="6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca" containerID="617866489a9ec9463a1170119181595771f4b1851be5c1778a5001066b1ae289" exitCode=0 Oct 29 22:15:59.847867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:59.847798 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" event={"ID":"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca","Type":"ContainerDied","Data":"617866489a9ec9463a1170119181595771f4b1851be5c1778a5001066b1ae289"} Oct 29 22:15:59.924699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:59.924663 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:59.980276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:15:59.980246 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:16:00.025410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.025381 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-maas-api-tls\") pod \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " Oct 29 22:16:00.025637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.025526 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-openshift-service-ca\") pod \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " Oct 29 22:16:00.025950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.025880 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-kube-api-access-wxp2t\" (UniqueName: \"kubernetes.io/projected/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-kube-api-access-wxp2t\") pod \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " Oct 29 22:16:00.026121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.026058 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-metrics-certs\") pod \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\" (UID: \"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca\") " Oct 29 22:16:00.026187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.026140 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-openshift-service-ca" pod "6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca" (UID: "6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:00.026946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.026924 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.034378 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.034350 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-maas-api-tls" pod "6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca" (UID: "6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.036093 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.036066 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-metrics-certs" pod "6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca" (UID: "6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.036394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.036374 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-kube-api-access-wxp2t" pod "6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca" (UID: "6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca"). InnerVolumeSpecName "kube-api-access-wxp2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:00.123971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.123868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:00.129033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.129002 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxp2t\" (UniqueName: \"kubernetes.io/projected/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-kube-api-access-wxp2t\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.129033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.129037 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.129203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.129051 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.298490 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.298450 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:00.675070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.675038 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:00.860638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.860600 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" Oct 29 22:16:00.860839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.860594 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2" event={"ID":"6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca","Type":"ContainerDied","Data":"778ef0e01c6ecf23d07e59fb9e213434fd9852b8531f9edfd9e552500036eeaf"} Oct 29 22:16:00.860839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.860742 4952 scope.go:122] "RemoveContainer" containerID="617866489a9ec9463a1170119181595771f4b1851be5c1778a5001066b1ae289" Oct 29 22:16:00.916348 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.916272 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2"] Oct 29 22:16:00.928192 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:00.928164 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-971004-b44bd4f7-nk8x2"] Oct 29 22:16:02.366003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:02.359841 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:02.665588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:02.665497 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca" path="/var/lib/kubelet/pods/6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca/volumes" Oct 29 22:16:04.031776 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:04.031520 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:04.268423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:04.268390 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:16:04.281164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:04.281136 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:16:04.963444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:04.963414 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:16:05.703308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:05.703225 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:07.423727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:07.423687 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:07.590927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:07.587217 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:16:08.281932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:08.273991 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:08.444268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:08.444231 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:08.619350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:08.619319 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:09.288656 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.288620 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:09.547579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.547494 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:09.781811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.781769 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:09.903537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.903504 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9"] Oct 29 22:16:09.923509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.904798 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9825f391-5b6d-4575-820f-62febeeb352b" containerName="main" Oct 29 22:16:09.938275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.938245 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9825f391-5b6d-4575-820f-62febeeb352b" containerName="main" Oct 29 22:16:09.938588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.938571 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca" containerName="maas-api" Oct 29 22:16:09.943136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.943112 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca" containerName="maas-api" Oct 29 22:16:09.950209 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.950167 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9825f391-5b6d-4575-820f-62febeeb352b" containerName="main" Oct 29 22:16:09.950481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.950464 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c02ac7c-4f2a-4a8e-9ae2-bff97b57dfca" containerName="maas-api" Oct 29 22:16:09.957134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.957111 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9"] Oct 29 22:16:09.960082 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:09.957806 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.025116 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.010455 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-bf99f9-openshift-default-dockercfg-nd6x5\"" Oct 29 22:16:10.187109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.186444 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:10.192963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.188432 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-data\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.192963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.188588 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-envoy\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.192963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.188633 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdcf82c-0135-4073-a97e-63dd77911285-istio-podinfo\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.192963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.188666 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdcf82c-0135-4073-a97e-63dd77911285-istiod-ca-cert\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.192963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.188732 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-credential-socket\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.192963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.188826 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dhn\" (UniqueName: \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-kube-api-access-l8dhn\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.192963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.188941 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-socket\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.192963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.188980 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-istio-token\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.192963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.189114 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-certs\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.299927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.298618 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-credential-socket\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.299927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.298716 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dhn\" (UniqueName: \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-kube-api-access-l8dhn\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.299927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.298801 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-socket\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.299927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.298880 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-istio-token\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.299927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.299073 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-credential-socket\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.300362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.300252 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-certs\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.300362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.300304 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-data\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.300473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.300422 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-envoy\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.300473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.300458 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdcf82c-0135-4073-a97e-63dd77911285-istio-podinfo\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.300610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.300486 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdcf82c-0135-4073-a97e-63dd77911285-istiod-ca-cert\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.301318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.301285 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-certs\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.301454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.301353 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdcf82c-0135-4073-a97e-63dd77911285-istiod-ca-cert\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.301620 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.301598 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-data\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.301965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.301939 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-socket\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.305195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.305167 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-envoy\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.305311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.305224 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdcf82c-0135-4073-a97e-63dd77911285-istio-podinfo\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.328829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.328780 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-istio-token\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.330640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.330608 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dhn\" (UniqueName: \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-kube-api-access-l8dhn\") pod \"e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.547009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.541049 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:10.558944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.552822 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:10.916103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.916071 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9"] Oct 29 22:16:10.919459 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:16:10.919433 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdcf82c_0135_4073_a97e_63dd77911285.slice/crio-dfc9d337fa2bbae0cff02efd68defc7913f7f30a98a26ad9f2d39d5ef1c877f8 WatchSource:0}: Error finding container dfc9d337fa2bbae0cff02efd68defc7913f7f30a98a26ad9f2d39d5ef1c877f8: Status 404 returned error can't find the container with id dfc9d337fa2bbae0cff02efd68defc7913f7f30a98a26ad9f2d39d5ef1c877f8 Oct 29 22:16:10.923451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.923408 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:16:10.923575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.923487 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:16:10.923575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:10.923518 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:16:11.284012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:11.283262 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" event={"ID":"5fdcf82c-0135-4073-a97e-63dd77911285","Type":"ContainerStarted","Data":"390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1"} Oct 29 22:16:11.284012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:11.283331 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" event={"ID":"5fdcf82c-0135-4073-a97e-63dd77911285","Type":"ContainerStarted","Data":"dfc9d337fa2bbae0cff02efd68defc7913f7f30a98a26ad9f2d39d5ef1c877f8"} Oct 29 22:16:11.355530 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:11.351724 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" podStartSLOduration=2.351704773 podStartE2EDuration="2.351704773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:11.343025975 +0000 UTC m=+2036.817614394" watchObservedRunningTime="2026-10-29 22:16:11.351704773 +0000 UTC m=+2036.826293192" Oct 29 22:16:11.555110 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:11.555013 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:11.668562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:11.668526 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:11.841294 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:11.841251 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:12.120865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:12.120744 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:12.561056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:12.560965 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:13.305008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:13.304943 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:13.306160 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:13.306136 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:13.745486 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:13.745448 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:15.468925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:15.465506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:15.550660 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:16:15.550620 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:16:16.073728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.073699 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t"] Oct 29 22:16:16.092271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.092168 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.113653 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.113623 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t"] Oct 29 22:16:16.118700 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.118675 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-675bf4\"/\"openshift-service-ca.crt\"" Oct 29 22:16:16.118857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.118815 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-675bf4\"/\"default-dockercfg-mnb7j\"" Oct 29 22:16:16.120693 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.120673 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-675bf4\"/\"kube-root-ca.crt\"" Oct 29 22:16:16.120867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.120718 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-675bf4\"/\"auth-test-model-675bf4-kserve-self-signed-certs\"" Oct 29 22:16:16.145588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.145221 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tls-certs\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.145588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.145334 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-dshm\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.145588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.145416 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-model-cache\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.145885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.145614 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-home\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.145885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.145824 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92csv\" (UniqueName: \"kubernetes.io/projected/9675baec-0ef5-4f3b-a9a0-be24996fb45f-kube-api-access-92csv\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.145885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.145869 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tmp-dir\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.210038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.209998 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:16.247385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.247344 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-model-cache\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.247569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.247515 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-home\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.247672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.247656 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92csv\" (UniqueName: \"kubernetes.io/projected/9675baec-0ef5-4f3b-a9a0-be24996fb45f-kube-api-access-92csv\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.247749 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.247704 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tmp-dir\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.248003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.247831 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tls-certs\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.248003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.247886 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-home\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.248003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.247959 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-dshm\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.248605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.248577 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tmp-dir\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.248799 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.248782 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-model-cache\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.250369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.250348 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-dshm\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.252980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.252089 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tls-certs\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.273616 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.273504 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92csv\" (UniqueName: \"kubernetes.io/projected/9675baec-0ef5-4f3b-a9a0-be24996fb45f-kube-api-access-92csv\") pod \"auth-test-model-675bf4-kserve-59b49f5b5-7gl5t\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.434976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.434433 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:16.915394 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:16:16.915362 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9675baec_0ef5_4f3b_a9a0_be24996fb45f.slice/crio-2ed3d926f78e1163a6eaaa80859e27748ab6d21b262ab7f4cc523df9ac5e1156 WatchSource:0}: Error finding container 2ed3d926f78e1163a6eaaa80859e27748ab6d21b262ab7f4cc523df9ac5e1156: Status 404 returned error can't find the container with id 2ed3d926f78e1163a6eaaa80859e27748ab6d21b262ab7f4cc523df9ac5e1156 Oct 29 22:16:16.915720 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:16.915415 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t"] Oct 29 22:16:17.380762 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:17.380721 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:17.422971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:17.422928 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" event={"ID":"9675baec-0ef5-4f3b-a9a0-be24996fb45f","Type":"ContainerStarted","Data":"e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b"} Oct 29 22:16:17.423161 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:17.422988 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" event={"ID":"9675baec-0ef5-4f3b-a9a0-be24996fb45f","Type":"ContainerStarted","Data":"2ed3d926f78e1163a6eaaa80859e27748ab6d21b262ab7f4cc523df9ac5e1156"} Oct 29 22:16:17.435133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:17.435101 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:17.470195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:17.470136 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" podStartSLOduration=1.47012055 podStartE2EDuration="1.47012055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:17.464877723 +0000 UTC m=+2042.939466142" watchObservedRunningTime="2026-10-29 22:16:17.47012055 +0000 UTC m=+2042.944709042" Oct 29 22:16:19.049926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:19.042565 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:20.771974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:20.765865 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:22.333789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.333750 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7"] Oct 29 22:16:22.338877 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.338847 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.367602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.367566 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7"] Oct 29 22:16:22.376871 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.370040 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-798a8f-serving-cert\"" Oct 29 22:16:22.422604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.422568 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.422959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.422930 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1deeeb21-199d-4f40-81c6-192ce3dbb82a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.423106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.423073 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.423286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.423265 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4625q\" (UniqueName: \"kubernetes.io/projected/1deeeb21-199d-4f40-81c6-192ce3dbb82a-kube-api-access-4625q\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.475998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.475953 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:22.524696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.524656 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1deeeb21-199d-4f40-81c6-192ce3dbb82a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.524882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.524773 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.525016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.524974 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4625q\" (UniqueName: \"kubernetes.io/projected/1deeeb21-199d-4f40-81c6-192ce3dbb82a-kube-api-access-4625q\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.525137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.525121 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.525474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.525451 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1deeeb21-199d-4f40-81c6-192ce3dbb82a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.527495 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.527458 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.527787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.527762 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.551198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.551166 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4625q\" (UniqueName: \"kubernetes.io/projected/1deeeb21-199d-4f40-81c6-192ce3dbb82a-kube-api-access-4625q\") pod \"maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:22.651936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.648893 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:22.683935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:22.683881 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:23.012675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:23.012643 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7"] Oct 29 22:16:23.013053 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:16:23.013029 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1deeeb21_199d_4f40_81c6_192ce3dbb82a.slice/crio-490b0ac572d2844e814b3a0ee1c0c3587a33cec99789f5244f6fb89d07758f6f WatchSource:0}: Error finding container 490b0ac572d2844e814b3a0ee1c0c3587a33cec99789f5244f6fb89d07758f6f: Status 404 returned error can't find the container with id 490b0ac572d2844e814b3a0ee1c0c3587a33cec99789f5244f6fb89d07758f6f Oct 29 22:16:23.483225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:23.483179 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" event={"ID":"1deeeb21-199d-4f40-81c6-192ce3dbb82a","Type":"ContainerStarted","Data":"490b0ac572d2844e814b3a0ee1c0c3587a33cec99789f5244f6fb89d07758f6f"} Oct 29 22:16:24.494684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:24.494645 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" event={"ID":"1deeeb21-199d-4f40-81c6-192ce3dbb82a","Type":"ContainerStarted","Data":"4faf8451d4431ab1fc7719c5171dd069e852389bdea8872106aae6d5128feb92"} Oct 29 22:16:24.495140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:24.494740 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:24.540533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:24.540474 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" podStartSLOduration=2.085975474 podStartE2EDuration="2.540456193s" totalImagesPullingTime="454.480719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:24.536853974 +0000 UTC m=+2050.011442394" watchObservedRunningTime="2026-10-29 22:16:24.540456193 +0000 UTC m=+2050.015044611" Oct 29 22:16:25.436296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:25.436253 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:26.265085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:26.265051 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:26.435557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:26.435521 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:26.448134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:26.448105 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:26.521801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:26.521727 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:16:27.857308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:27.857270 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:29.322403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.322366 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9"] Oct 29 22:16:29.328712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.328682 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.353870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.351929 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-bf99f9\"/\"default-dockercfg-xl299\"" Oct 29 22:16:29.354081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.353525 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-bf99f9\"/\"noten-model-bf99f9-kserve-self-signed-certs\"" Oct 29 22:16:29.363414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.363393 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-bf99f9\"/\"kube-root-ca.crt\"" Oct 29 22:16:29.371925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.363633 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-bf99f9\"/\"openshift-service-ca.crt\"" Oct 29 22:16:29.374684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.374657 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9"] Oct 29 22:16:29.458271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.458227 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:29.472594 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.472559 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ec6317-d0e6-4a56-bdf5-8143739a20df-tls-certs\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.472770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.472614 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-dshm\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.472770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.472708 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-home\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.472859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.472837 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5w5\" (UniqueName: \"kubernetes.io/projected/75ec6317-d0e6-4a56-bdf5-8143739a20df-kube-api-access-4c5w5\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.473076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.473062 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-tmp-dir\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.473140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.473126 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-model-cache\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.597013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.591957 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-tmp-dir\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.597013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.592077 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-model-cache\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.597013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.592204 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ec6317-d0e6-4a56-bdf5-8143739a20df-tls-certs\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.597013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.592261 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-dshm\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.597013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.592288 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-home\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.597013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.592365 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5w5\" (UniqueName: \"kubernetes.io/projected/75ec6317-d0e6-4a56-bdf5-8143739a20df-kube-api-access-4c5w5\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.597013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.593514 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-tmp-dir\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.597013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.593852 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-model-cache\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.599045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.599008 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-home\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.630636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.630570 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5w5\" (UniqueName: \"kubernetes.io/projected/75ec6317-d0e6-4a56-bdf5-8143739a20df-kube-api-access-4c5w5\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.640926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.636474 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ec6317-d0e6-4a56-bdf5-8143739a20df-tls-certs\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.656515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.656480 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-dshm\") pod \"noten-model-bf99f9-kserve-5766b8f9f7-rp6t9\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.678523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.678219 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:29.718433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:29.718396 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:30.243115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:30.243082 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9"] Oct 29 22:16:30.243304 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:16:30.243279 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75ec6317_d0e6_4a56_bdf5_8143739a20df.slice/crio-37b9f44f771609d56668b4615730bff792936ff4bc5000277f1ff990bd9e02c9 WatchSource:0}: Error finding container 37b9f44f771609d56668b4615730bff792936ff4bc5000277f1ff990bd9e02c9: Status 404 returned error can't find the container with id 37b9f44f771609d56668b4615730bff792936ff4bc5000277f1ff990bd9e02c9 Oct 29 22:16:30.507448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:30.507421 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:16:30.555863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:30.555752 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" event={"ID":"75ec6317-d0e6-4a56-bdf5-8143739a20df","Type":"ContainerStarted","Data":"c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef"} Oct 29 22:16:30.555863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:30.555817 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" event={"ID":"75ec6317-d0e6-4a56-bdf5-8143739a20df","Type":"ContainerStarted","Data":"37b9f44f771609d56668b4615730bff792936ff4bc5000277f1ff990bd9e02c9"} Oct 29 22:16:30.612509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:30.612443 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" podStartSLOduration=1.6124234670000002 podStartE2EDuration="1.612423467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:30.599147844 +0000 UTC m=+2056.073736274" watchObservedRunningTime="2026-10-29 22:16:30.612423467 +0000 UTC m=+2056.087011892" Oct 29 22:16:30.679094 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:30.679056 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:31.655929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:31.650761 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:33.428938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:33.424890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:35.112239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:35.112201 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:36.828670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:36.828607 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:38.597931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:38.595696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:39.679483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:39.679442 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:39.692524 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:39.692493 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:39.725505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:39.725469 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:16:40.444936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:40.443826 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:42.132494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:42.129352 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:43.009929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:43.007392 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:43.847828 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:43.844184 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:45.352099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:45.352066 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:45.534804 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:16:45.534763 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:16:45.654764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:45.654674 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:45.885198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:45.884539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:46.276655 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:46.276524 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:47.558079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:47.557971 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:49.464421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:49.464384 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:49.718239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:49.718134 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:49.897390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:49.897344 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:50.981080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:50.981039 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:51.406360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:51.406319 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq"] Oct 29 22:16:51.407325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:51.407285 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerName="istio-proxy" containerID="cri-o://69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1" gracePeriod=30 Oct 29 22:16:51.803514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:51.803413 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:52.398129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:52.398090 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:52.593743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:52.593702 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9"] Oct 29 22:16:52.860011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:52.859970 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:53.383960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:53.358000 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9"] Oct 29 22:16:53.383960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:53.358265 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" podUID="5fdcf82c-0135-4073-a97e-63dd77911285" containerName="istio-proxy" containerID="cri-o://390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1" gracePeriod=30 Oct 29 22:16:53.433509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:53.433372 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:53.615736 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:53.615704 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:53.727062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:53.725578 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:16:54.328548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:54.328514 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:56.413670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.413564 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:56.768878 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.768842 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:16:56.944128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.944093 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-kube-api-access-4wf5k\" (UniqueName: \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-kube-api-access-4wf5k\") pod \"2658a91b-fbc8-4894-8ce9-3ba063792034\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " Oct 29 22:16:56.944340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.944223 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-podinfo\") pod \"2658a91b-fbc8-4894-8ce9-3ba063792034\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " Oct 29 22:16:56.944340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.944327 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-socket\") pod \"2658a91b-fbc8-4894-8ce9-3ba063792034\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " Oct 29 22:16:56.944539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.944431 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-envoy\") pod \"2658a91b-fbc8-4894-8ce9-3ba063792034\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " Oct 29 22:16:56.945584 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.945537 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-socket" pod "2658a91b-fbc8-4894-8ce9-3ba063792034" (UID: "2658a91b-fbc8-4894-8ce9-3ba063792034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:56.949410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.944555 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2658a91b-fbc8-4894-8ce9-3ba063792034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2658a91b-fbc8-4894-8ce9-3ba063792034-istiod-ca-cert\") pod \"2658a91b-fbc8-4894-8ce9-3ba063792034\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " Oct 29 22:16:56.949738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.949647 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-token\" (UniqueName: \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-token\") pod \"2658a91b-fbc8-4894-8ce9-3ba063792034\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " Oct 29 22:16:56.952281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.950327 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-certs\") pod \"2658a91b-fbc8-4894-8ce9-3ba063792034\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " Oct 29 22:16:56.952281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.950470 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-data\") pod \"2658a91b-fbc8-4894-8ce9-3ba063792034\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " Oct 29 22:16:56.952281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.950536 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-credential-socket\") pod \"2658a91b-fbc8-4894-8ce9-3ba063792034\" (UID: \"2658a91b-fbc8-4894-8ce9-3ba063792034\") " Oct 29 22:16:56.953121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.952858 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2658a91b-fbc8-4894-8ce9-3ba063792034-istiod-ca-cert" pod "2658a91b-fbc8-4894-8ce9-3ba063792034" (UID: "2658a91b-fbc8-4894-8ce9-3ba063792034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:56.953743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.953669 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-certs" pod "2658a91b-fbc8-4894-8ce9-3ba063792034" (UID: "2658a91b-fbc8-4894-8ce9-3ba063792034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:56.954174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.954100 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-data" pod "2658a91b-fbc8-4894-8ce9-3ba063792034" (UID: "2658a91b-fbc8-4894-8ce9-3ba063792034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:56.954709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.954677 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-credential-socket" pod "2658a91b-fbc8-4894-8ce9-3ba063792034" (UID: "2658a91b-fbc8-4894-8ce9-3ba063792034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:56.955603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.955578 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2658a91b-fbc8-4894-8ce9-3ba063792034-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:56.955728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.955607 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:56.955728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.955622 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:56.955728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.955633 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:56.955728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.955646 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:56.956073 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.956048 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-podinfo" pod "2658a91b-fbc8-4894-8ce9-3ba063792034" (UID: "2658a91b-fbc8-4894-8ce9-3ba063792034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:16:56.958290 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.958263 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-kube-api-access-4wf5k" pod "2658a91b-fbc8-4894-8ce9-3ba063792034" (UID: "2658a91b-fbc8-4894-8ce9-3ba063792034"). InnerVolumeSpecName "kube-api-access-4wf5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:56.960636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.960588 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-envoy" pod "2658a91b-fbc8-4894-8ce9-3ba063792034" (UID: "2658a91b-fbc8-4894-8ce9-3ba063792034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:56.965436 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:56.965380 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-token" pod "2658a91b-fbc8-4894-8ce9-3ba063792034" (UID: "2658a91b-fbc8-4894-8ce9-3ba063792034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:57.058236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.058188 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wf5k\" (UniqueName: \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-kube-api-access-4wf5k\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:57.058236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.058231 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:57.058603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.058246 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:57.058603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.058271 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2658a91b-fbc8-4894-8ce9-3ba063792034-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:57.135680 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.135637 4952 generic.go:356] "Generic (PLEG): container finished" podID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerID="69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1" exitCode=0 Oct 29 22:16:57.135889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.135705 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" Oct 29 22:16:57.135889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.135752 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" event={"ID":"2658a91b-fbc8-4894-8ce9-3ba063792034","Type":"ContainerDied","Data":"69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1"} Oct 29 22:16:57.135889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.135849 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq" event={"ID":"2658a91b-fbc8-4894-8ce9-3ba063792034","Type":"ContainerDied","Data":"cab1656a305a69004ddd7faa4377073f1354d03f1a8d2cc48803f7c95cebfa9e"} Oct 29 22:16:57.135889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.135878 4952 scope.go:122] "RemoveContainer" containerID="69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1" Oct 29 22:16:57.165826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.165796 4952 scope.go:122] "RemoveContainer" containerID="69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1" Oct 29 22:16:57.166276 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:16:57.166247 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1\": container with ID starting with 69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1 not found: ID does not exist" containerID="69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1" Oct 29 22:16:57.166372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.166288 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1"} err="failed to get container status \"69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1\": rpc error: code = NotFound desc = could not find container \"69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1\": container with ID starting with 69f40342ab683496fd4b64fa7308d19870bdb40d97fe2214732dccccdce941f1 not found: ID does not exist" Oct 29 22:16:57.204527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.204486 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq"] Oct 29 22:16:57.209748 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:57.209717 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-gtsbq"] Oct 29 22:16:58.193812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.193773 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:58.310961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.308498 4952 patch_prober.go:29] interesting pod/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:16:58.310961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.308602 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" podUID="5fdcf82c-0135-4073-a97e-63dd77911285" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:16:58.401253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.401200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:58.656529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.656485 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" path="/var/lib/kubelet/pods/2658a91b-fbc8-4894-8ce9-3ba063792034/volumes" Oct 29 22:16:58.806711 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.806642 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:58.896623 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.896585 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-kube-api-access-l8dhn\" (UniqueName: \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-kube-api-access-l8dhn\") pod \"5fdcf82c-0135-4073-a97e-63dd77911285\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " Oct 29 22:16:58.896792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.896647 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-envoy\") pod \"5fdcf82c-0135-4073-a97e-63dd77911285\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " Oct 29 22:16:58.896792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.896693 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-credential-socket\") pod \"5fdcf82c-0135-4073-a97e-63dd77911285\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " Oct 29 22:16:58.897010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.896983 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-istio-token\" (UniqueName: \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-istio-token\") pod \"5fdcf82c-0135-4073-a97e-63dd77911285\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " Oct 29 22:16:58.897151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.897022 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-socket\") pod \"5fdcf82c-0135-4073-a97e-63dd77911285\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " Oct 29 22:16:58.897151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.897098 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-certs\") pod \"5fdcf82c-0135-4073-a97e-63dd77911285\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " Oct 29 22:16:58.897295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.897202 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fdcf82c-0135-4073-a97e-63dd77911285-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdcf82c-0135-4073-a97e-63dd77911285-istiod-ca-cert\") pod \"5fdcf82c-0135-4073-a97e-63dd77911285\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " Oct 29 22:16:58.897295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.897275 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-data\") pod \"5fdcf82c-0135-4073-a97e-63dd77911285\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " Oct 29 22:16:58.897406 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.897301 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fdcf82c-0135-4073-a97e-63dd77911285-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdcf82c-0135-4073-a97e-63dd77911285-istio-podinfo\") pod \"5fdcf82c-0135-4073-a97e-63dd77911285\" (UID: \"5fdcf82c-0135-4073-a97e-63dd77911285\") " Oct 29 22:16:58.898319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.898245 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-credential-socket" pod "5fdcf82c-0135-4073-a97e-63dd77911285" (UID: "5fdcf82c-0135-4073-a97e-63dd77911285"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:58.898319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.898263 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-socket" pod "5fdcf82c-0135-4073-a97e-63dd77911285" (UID: "5fdcf82c-0135-4073-a97e-63dd77911285"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:58.898855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.898471 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-data" pod "5fdcf82c-0135-4073-a97e-63dd77911285" (UID: "5fdcf82c-0135-4073-a97e-63dd77911285"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:58.898855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.898481 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fdcf82c-0135-4073-a97e-63dd77911285-istiod-ca-cert" pod "5fdcf82c-0135-4073-a97e-63dd77911285" (UID: "5fdcf82c-0135-4073-a97e-63dd77911285"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:58.898855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.898721 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:58.898855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.898744 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:58.898855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.898759 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fdcf82c-0135-4073-a97e-63dd77911285-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:58.898855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.898773 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:58.901167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.900028 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-certs" pod "5fdcf82c-0135-4073-a97e-63dd77911285" (UID: "5fdcf82c-0135-4073-a97e-63dd77911285"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:58.901608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.901570 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fdcf82c-0135-4073-a97e-63dd77911285-istio-podinfo" pod "5fdcf82c-0135-4073-a97e-63dd77911285" (UID: "5fdcf82c-0135-4073-a97e-63dd77911285"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:16:58.901704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.901602 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-kube-api-access-l8dhn" pod "5fdcf82c-0135-4073-a97e-63dd77911285" (UID: "5fdcf82c-0135-4073-a97e-63dd77911285"). InnerVolumeSpecName "kube-api-access-l8dhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:58.902252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.902207 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-istio-token" pod "5fdcf82c-0135-4073-a97e-63dd77911285" (UID: "5fdcf82c-0135-4073-a97e-63dd77911285"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:58.903281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:58.903225 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-envoy" pod "5fdcf82c-0135-4073-a97e-63dd77911285" (UID: "5fdcf82c-0135-4073-a97e-63dd77911285"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:59.000310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.000265 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:59.000310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.000302 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:59.000310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.000316 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf82c-0135-4073-a97e-63dd77911285-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:59.000561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.000329 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fdcf82c-0135-4073-a97e-63dd77911285-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:59.000561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.000341 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8dhn\" (UniqueName: \"kubernetes.io/projected/5fdcf82c-0135-4073-a97e-63dd77911285-kube-api-access-l8dhn\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:16:59.158520 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.158479 4952 generic.go:356] "Generic (PLEG): container finished" podID="5fdcf82c-0135-4073-a97e-63dd77911285" containerID="390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1" exitCode=0 Oct 29 22:16:59.158714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.158527 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" Oct 29 22:16:59.158714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.158558 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" event={"ID":"5fdcf82c-0135-4073-a97e-63dd77911285","Type":"ContainerDied","Data":"390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1"} Oct 29 22:16:59.158714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.158631 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9" event={"ID":"5fdcf82c-0135-4073-a97e-63dd77911285","Type":"ContainerDied","Data":"dfc9d337fa2bbae0cff02efd68defc7913f7f30a98a26ad9f2d39d5ef1c877f8"} Oct 29 22:16:59.158714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.158662 4952 scope.go:122] "RemoveContainer" containerID="390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1" Oct 29 22:16:59.174098 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.174074 4952 scope.go:122] "RemoveContainer" containerID="390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1" Oct 29 22:16:59.174391 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:16:59.174372 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1\": container with ID starting with 390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1 not found: ID does not exist" containerID="390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1" Oct 29 22:16:59.174489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.174398 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1"} err="failed to get container status \"390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1\": rpc error: code = NotFound desc = could not find container \"390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1\": container with ID starting with 390031df6bffbf27d3ee7df999fb79fc622c4f8def10c3bbecedc782091465a1 not found: ID does not exist" Oct 29 22:16:59.214648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.214617 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9"] Oct 29 22:16:59.227006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.226963 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-bf99f9-openshift-default-868cdfcfc9-k89g9"] Oct 29 22:16:59.984995 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:16:59.984285 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:00.649930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:00.649879 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fdcf82c-0135-4073-a97e-63dd77911285" path="/var/lib/kubelet/pods/5fdcf82c-0135-4073-a97e-63dd77911285/volumes" Oct 29 22:17:02.067931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:02.063880 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:03.495303 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:03.494673 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:03.840965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:03.840928 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:04.345519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.345485 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-9450e4"] Oct 29 22:17:04.346382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.346356 4952 cpu_manager.go:414] "containerMap: removing container" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerName="istio-proxy" Oct 29 22:17:04.346550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.346527 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerName="istio-proxy" Oct 29 22:17:04.346702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.346574 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5fdcf82c-0135-4073-a97e-63dd77911285" containerName="istio-proxy" Oct 29 22:17:04.346702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.346583 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdcf82c-0135-4073-a97e-63dd77911285" containerName="istio-proxy" Oct 29 22:17:04.346854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.346833 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="2658a91b-fbc8-4894-8ce9-3ba063792034" containerName="istio-proxy" Oct 29 22:17:04.346854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.346854 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdcf82c-0135-4073-a97e-63dd77911285" containerName="istio-proxy" Oct 29 22:17:04.352796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.352768 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9450e4" Oct 29 22:17:04.366018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.365987 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-9450e4"] Oct 29 22:17:04.372247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.372217 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r84ws\"" Oct 29 22:17:04.539021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.538979 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7lg\" (UniqueName: \"kubernetes.io/projected/a843f495-6b86-44b6-af22-c8bd40b98d43-kube-api-access-zj7lg\") pod \"test-curl-7761-9450e4\" (UID: \"a843f495-6b86-44b6-af22-c8bd40b98d43\") " pod="opendatahub/test-curl-7761-9450e4" Oct 29 22:17:04.640506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.640410 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7lg\" (UniqueName: \"kubernetes.io/projected/a843f495-6b86-44b6-af22-c8bd40b98d43-kube-api-access-zj7lg\") pod \"test-curl-7761-9450e4\" (UID: \"a843f495-6b86-44b6-af22-c8bd40b98d43\") " pod="opendatahub/test-curl-7761-9450e4" Oct 29 22:17:04.667265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.667159 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7lg\" (UniqueName: \"kubernetes.io/projected/a843f495-6b86-44b6-af22-c8bd40b98d43-kube-api-access-zj7lg\") pod \"test-curl-7761-9450e4\" (UID: \"a843f495-6b86-44b6-af22-c8bd40b98d43\") " pod="opendatahub/test-curl-7761-9450e4" Oct 29 22:17:04.673101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:04.673065 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9450e4" Oct 29 22:17:05.015056 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:17:05.015022 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda843f495_6b86_44b6_af22_c8bd40b98d43.slice/crio-eacfab0ae27d694e8b8e0d3055f28a3f36a96db702d2a5788d88339821d9ed3e WatchSource:0}: Error finding container eacfab0ae27d694e8b8e0d3055f28a3f36a96db702d2a5788d88339821d9ed3e: Status 404 returned error can't find the container with id eacfab0ae27d694e8b8e0d3055f28a3f36a96db702d2a5788d88339821d9ed3e Oct 29 22:17:05.015474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:05.015398 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-9450e4"] Oct 29 22:17:05.211990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:05.211943 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-9450e4" event={"ID":"a843f495-6b86-44b6-af22-c8bd40b98d43","Type":"ContainerStarted","Data":"6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73"} Oct 29 22:17:05.212187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:05.212004 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-9450e4" event={"ID":"a843f495-6b86-44b6-af22-c8bd40b98d43","Type":"ContainerStarted","Data":"eacfab0ae27d694e8b8e0d3055f28a3f36a96db702d2a5788d88339821d9ed3e"} Oct 29 22:17:05.254398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:05.254337 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7761-9450e4" podStartSLOduration=1.254322289 podStartE2EDuration="1.254322289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:05.250740536 +0000 UTC m=+2090.725328954" watchObservedRunningTime="2026-10-29 22:17:05.254322289 +0000 UTC m=+2090.728910707" Oct 29 22:17:05.660306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:05.660267 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:06.117568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.117529 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-9450e4"] Oct 29 22:17:06.123257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.123220 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-9450e4"] Oct 29 22:17:06.280835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.280796 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-cd455e"] Oct 29 22:17:06.281509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.281493 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a843f495-6b86-44b6-af22-c8bd40b98d43" containerName="test-curl-7761-9450e4" Oct 29 22:17:06.281560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.281510 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a843f495-6b86-44b6-af22-c8bd40b98d43" containerName="test-curl-7761-9450e4" Oct 29 22:17:06.281658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.281648 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a843f495-6b86-44b6-af22-c8bd40b98d43" containerName="test-curl-7761-9450e4" Oct 29 22:17:06.285179 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.285160 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-cd455e" Oct 29 22:17:06.320016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.319977 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-cd455e"] Oct 29 22:17:06.381962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.381801 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzjbp\" (UniqueName: \"kubernetes.io/projected/61bd515f-80cd-4e6e-95d8-38db04cc350e-kube-api-access-tzjbp\") pod \"test-curl-7761-cd455e\" (UID: \"61bd515f-80cd-4e6e-95d8-38db04cc350e\") " pod="opendatahub/test-curl-7761-cd455e" Oct 29 22:17:06.483576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.483536 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzjbp\" (UniqueName: \"kubernetes.io/projected/61bd515f-80cd-4e6e-95d8-38db04cc350e-kube-api-access-tzjbp\") pod \"test-curl-7761-cd455e\" (UID: \"61bd515f-80cd-4e6e-95d8-38db04cc350e\") " pod="opendatahub/test-curl-7761-cd455e" Oct 29 22:17:06.508356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.508318 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzjbp\" (UniqueName: \"kubernetes.io/projected/61bd515f-80cd-4e6e-95d8-38db04cc350e-kube-api-access-tzjbp\") pod \"test-curl-7761-cd455e\" (UID: \"61bd515f-80cd-4e6e-95d8-38db04cc350e\") " pod="opendatahub/test-curl-7761-cd455e" Oct 29 22:17:06.598119 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.598084 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-cd455e" Oct 29 22:17:06.924189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:06.924162 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-cd455e"] Oct 29 22:17:06.927256 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:17:06.927188 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61bd515f_80cd_4e6e_95d8_38db04cc350e.slice/crio-7d40cef2b0b28cc178b7288ddee71b00be72ed7693eeaa7cc47672da196822d7 WatchSource:0}: Error finding container 7d40cef2b0b28cc178b7288ddee71b00be72ed7693eeaa7cc47672da196822d7: Status 404 returned error can't find the container with id 7d40cef2b0b28cc178b7288ddee71b00be72ed7693eeaa7cc47672da196822d7 Oct 29 22:17:07.232633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.232525 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-cd455e" event={"ID":"61bd515f-80cd-4e6e-95d8-38db04cc350e","Type":"ContainerStarted","Data":"ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25"} Oct 29 22:17:07.232633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.232579 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-cd455e" event={"ID":"61bd515f-80cd-4e6e-95d8-38db04cc350e","Type":"ContainerStarted","Data":"7d40cef2b0b28cc178b7288ddee71b00be72ed7693eeaa7cc47672da196822d7"} Oct 29 22:17:07.232633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.232610 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7761-9450e4" podUID="a843f495-6b86-44b6-af22-c8bd40b98d43" containerName="test-curl-7761-9450e4" containerID="cri-o://6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73" gracePeriod=2 Oct 29 22:17:07.274502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.274443 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7761-cd455e" podStartSLOduration=1.274428588 podStartE2EDuration="1.274428588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:07.270710048 +0000 UTC m=+2092.745298467" watchObservedRunningTime="2026-10-29 22:17:07.274428588 +0000 UTC m=+2092.749017005" Oct 29 22:17:07.337691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.337614 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:07.622840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.622804 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" podUID="75ec6317-d0e6-4a56-bdf5-8143739a20df" containerName="main" containerID="cri-o://c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef" gracePeriod=45 Oct 29 22:17:07.713361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.713325 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" podUID="75ec6317-d0e6-4a56-bdf5-8143739a20df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.152:8000/ready\": dial tcp 10.132.0.152:8000: connect: connection refused" Oct 29 22:17:07.887088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.886994 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt"] Oct 29 22:17:07.906121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.906081 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:07.922892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.916889 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt"] Oct 29 22:17:07.984112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:07.984081 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:17:08.007106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.007078 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f52917-openshift-default-dockercfg-2q4hk\"" Oct 29 22:17:08.020045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020007 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-home\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-home\") pod \"75ec6317-d0e6-4a56-bdf5-8143739a20df\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " Oct 29 22:17:08.020184 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020058 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-model-cache\") pod \"75ec6317-d0e6-4a56-bdf5-8143739a20df\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " Oct 29 22:17:08.020233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020222 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ec6317-d0e6-4a56-bdf5-8143739a20df-tls-certs\" (UniqueName: \"kubernetes.io/secret/75ec6317-d0e6-4a56-bdf5-8143739a20df-tls-certs\") pod \"75ec6317-d0e6-4a56-bdf5-8143739a20df\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " Oct 29 22:17:08.020307 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020265 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-home" pod "75ec6317-d0e6-4a56-bdf5-8143739a20df" (UID: "75ec6317-d0e6-4a56-bdf5-8143739a20df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:08.020307 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020279 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-tmp-dir\") pod \"75ec6317-d0e6-4a56-bdf5-8143739a20df\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " Oct 29 22:17:08.020391 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020297 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-model-cache" pod "75ec6317-d0e6-4a56-bdf5-8143739a20df" (UID: "75ec6317-d0e6-4a56-bdf5-8143739a20df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:08.020452 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020435 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-dshm\") pod \"75ec6317-d0e6-4a56-bdf5-8143739a20df\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " Oct 29 22:17:08.020513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020493 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-tmp-dir" pod "75ec6317-d0e6-4a56-bdf5-8143739a20df" (UID: "75ec6317-d0e6-4a56-bdf5-8143739a20df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:08.020615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020599 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ec6317-d0e6-4a56-bdf5-8143739a20df-kube-api-access-4c5w5\" (UniqueName: \"kubernetes.io/projected/75ec6317-d0e6-4a56-bdf5-8143739a20df-kube-api-access-4c5w5\") pod \"75ec6317-d0e6-4a56-bdf5-8143739a20df\" (UID: \"75ec6317-d0e6-4a56-bdf5-8143739a20df\") " Oct 29 22:17:08.020866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020847 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.021017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.020991 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.021156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.021139 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.021218 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.021188 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tjw\" (UniqueName: \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-kube-api-access-k5tjw\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.021281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.021265 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.021342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.021286 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.021342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.021318 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.021449 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.021356 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.021449 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.021376 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44576234-7466-4ad4-9001-7c9b153fdcd3-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.021563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.021453 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:08.021563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.021464 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:08.021563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.021473 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:08.023024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.022993 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ec6317-d0e6-4a56-bdf5-8143739a20df-tls-certs" pod "75ec6317-d0e6-4a56-bdf5-8143739a20df" (UID: "75ec6317-d0e6-4a56-bdf5-8143739a20df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:17:08.023474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.023365 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ec6317-d0e6-4a56-bdf5-8143739a20df-kube-api-access-4c5w5" pod "75ec6317-d0e6-4a56-bdf5-8143739a20df" (UID: "75ec6317-d0e6-4a56-bdf5-8143739a20df"). InnerVolumeSpecName "kube-api-access-4c5w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:08.023474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.023439 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-dshm" pod "75ec6317-d0e6-4a56-bdf5-8143739a20df" (UID: "75ec6317-d0e6-4a56-bdf5-8143739a20df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:08.123463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.123421 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.123638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.123503 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.123638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.123630 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.123769 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.123695 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tjw\" (UniqueName: \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-kube-api-access-k5tjw\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.123840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.123803 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.123840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.123833 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.124003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.123876 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.124003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.123934 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.124003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.123953 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.124191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.124005 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44576234-7466-4ad4-9001-7c9b153fdcd3-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.124191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.124140 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ec6317-d0e6-4a56-bdf5-8143739a20df-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:08.124191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.124157 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5w5\" (UniqueName: \"kubernetes.io/projected/75ec6317-d0e6-4a56-bdf5-8143739a20df-kube-api-access-4c5w5\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:08.124191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.124174 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ec6317-d0e6-4a56-bdf5-8143739a20df-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:08.124191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.124002 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.124629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.124608 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.124727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.124648 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.124857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.124837 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44576234-7466-4ad4-9001-7c9b153fdcd3-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.126713 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.126686 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.127117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.127094 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.148591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.148525 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.149048 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.149027 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tjw\" (UniqueName: \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-kube-api-access-k5tjw\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.243621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.243577 4952 generic.go:356] "Generic (PLEG): container finished" podID="75ec6317-d0e6-4a56-bdf5-8143739a20df" containerID="c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef" exitCode=0 Oct 29 22:17:08.243797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.243692 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" Oct 29 22:17:08.243797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.243727 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" event={"ID":"75ec6317-d0e6-4a56-bdf5-8143739a20df","Type":"ContainerDied","Data":"c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef"} Oct 29 22:17:08.243797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.243781 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9" event={"ID":"75ec6317-d0e6-4a56-bdf5-8143739a20df","Type":"ContainerDied","Data":"37b9f44f771609d56668b4615730bff792936ff4bc5000277f1ff990bd9e02c9"} Oct 29 22:17:08.243921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.243805 4952 scope.go:122] "RemoveContainer" containerID="c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef" Oct 29 22:17:08.264229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.264141 4952 scope.go:122] "RemoveContainer" containerID="c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef" Oct 29 22:17:08.264970 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:08.264941 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef\": container with ID starting with c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef not found: ID does not exist" containerID="c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef" Oct 29 22:17:08.265080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.264981 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef"} err="failed to get container status \"c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef\": rpc error: code = NotFound desc = could not find container \"c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef\": container with ID starting with c8b15f9466910bab108be2113ed626ffe132e468b0c557c5b6b077a22d86c4ef not found: ID does not exist" Oct 29 22:17:08.287000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.286971 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:08.304413 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.304385 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-cd455e"] Oct 29 22:17:08.309992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.309964 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9"] Oct 29 22:17:08.318008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.314647 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-cd455e"] Oct 29 22:17:08.320261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.320233 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-bf99f9/noten-model-bf99f9-kserve-5766b8f9f7-rp6t9"] Oct 29 22:17:08.479963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.479864 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-54ff16"] Oct 29 22:17:08.481025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.480995 4952 cpu_manager.go:414] "containerMap: removing container" podUID="75ec6317-d0e6-4a56-bdf5-8143739a20df" containerName="main" Oct 29 22:17:08.481025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.481024 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ec6317-d0e6-4a56-bdf5-8143739a20df" containerName="main" Oct 29 22:17:08.481216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.481073 4952 cpu_manager.go:414] "containerMap: removing container" podUID="61bd515f-80cd-4e6e-95d8-38db04cc350e" containerName="test-curl-7761-cd455e" Oct 29 22:17:08.481216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.481081 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61bd515f-80cd-4e6e-95d8-38db04cc350e" containerName="test-curl-7761-cd455e" Oct 29 22:17:08.481369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.481319 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ec6317-d0e6-4a56-bdf5-8143739a20df" containerName="main" Oct 29 22:17:08.481369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.481346 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="61bd515f-80cd-4e6e-95d8-38db04cc350e" containerName="test-curl-7761-cd455e" Oct 29 22:17:08.485745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.485722 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:08.498346 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.498316 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-54ff16"] Oct 29 22:17:08.531814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.531780 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7xq\" (UniqueName: \"kubernetes.io/projected/98e4c782-3f72-46ea-97e9-e49f05bc5a37-kube-api-access-qm7xq\") pod \"test-curl-7761-54ff16\" (UID: \"98e4c782-3f72-46ea-97e9-e49f05bc5a37\") " pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:08.633664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.633625 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7xq\" (UniqueName: \"kubernetes.io/projected/98e4c782-3f72-46ea-97e9-e49f05bc5a37-kube-api-access-qm7xq\") pod \"test-curl-7761-54ff16\" (UID: \"98e4c782-3f72-46ea-97e9-e49f05bc5a37\") " pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:08.650843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.638465 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt"] Oct 29 22:17:08.650843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.650599 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:17:08.650843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.650683 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:17:08.650843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.650726 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:17:08.686847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.686813 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ec6317-d0e6-4a56-bdf5-8143739a20df" path="/var/lib/kubelet/pods/75ec6317-d0e6-4a56-bdf5-8143739a20df/volumes" Oct 29 22:17:08.691885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.691858 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7xq\" (UniqueName: \"kubernetes.io/projected/98e4c782-3f72-46ea-97e9-e49f05bc5a37-kube-api-access-qm7xq\") pod \"test-curl-7761-54ff16\" (UID: \"98e4c782-3f72-46ea-97e9-e49f05bc5a37\") " pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:08.803213 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:08.803189 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:09.159829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.159797 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-54ff16"] Oct 29 22:17:09.161704 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:17:09.161670 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98e4c782_3f72_46ea_97e9_e49f05bc5a37.slice/crio-49cd21fa57be54c43303213bded9bc4c0c6a438aaff16144a1b105e4b67e8f8a WatchSource:0}: Error finding container 49cd21fa57be54c43303213bded9bc4c0c6a438aaff16144a1b105e4b67e8f8a: Status 404 returned error can't find the container with id 49cd21fa57be54c43303213bded9bc4c0c6a438aaff16144a1b105e4b67e8f8a Oct 29 22:17:09.242935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.241512 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:09.289956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.285544 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" event={"ID":"44576234-7466-4ad4-9001-7c9b153fdcd3","Type":"ContainerStarted","Data":"9f3cab521de89ded609030fe41192da60e992b3286b71cf33d89c1b1813f79f0"} Oct 29 22:17:09.289956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.285610 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" event={"ID":"44576234-7466-4ad4-9001-7c9b153fdcd3","Type":"ContainerStarted","Data":"b56e152b9c712db1a7d828719829ebd1a042622fc82eeefb77356cdf00ec427e"} Oct 29 22:17:09.289956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.287263 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:09.290259 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.290057 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:17:09.290259 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.290108 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 29 22:17:09.290464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.290436 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-54ff16" event={"ID":"98e4c782-3f72-46ea-97e9-e49f05bc5a37","Type":"ContainerStarted","Data":"49cd21fa57be54c43303213bded9bc4c0c6a438aaff16144a1b105e4b67e8f8a"} Oct 29 22:17:09.294600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.294568 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7761-cd455e" podUID="61bd515f-80cd-4e6e-95d8-38db04cc350e" containerName="test-curl-7761-cd455e" containerID="cri-o://ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25" gracePeriod=2 Oct 29 22:17:09.344496 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.344427 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" podStartSLOduration=2.344409018 podStartE2EDuration="2.344409018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:09.336412859 +0000 UTC m=+2094.811001278" watchObservedRunningTime="2026-10-29 22:17:09.344409018 +0000 UTC m=+2094.818997438" Oct 29 22:17:09.529509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.529477 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9450e4" Oct 29 22:17:09.673346 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.673315 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a843f495-6b86-44b6-af22-c8bd40b98d43-kube-api-access-zj7lg\" (UniqueName: \"kubernetes.io/projected/a843f495-6b86-44b6-af22-c8bd40b98d43-kube-api-access-zj7lg\") pod \"a843f495-6b86-44b6-af22-c8bd40b98d43\" (UID: \"a843f495-6b86-44b6-af22-c8bd40b98d43\") " Oct 29 22:17:09.675546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.675519 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a843f495-6b86-44b6-af22-c8bd40b98d43-kube-api-access-zj7lg" pod "a843f495-6b86-44b6-af22-c8bd40b98d43" (UID: "a843f495-6b86-44b6-af22-c8bd40b98d43"). InnerVolumeSpecName "kube-api-access-zj7lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:09.718863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.718828 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:09.776251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:09.775813 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj7lg\" (UniqueName: \"kubernetes.io/projected/a843f495-6b86-44b6-af22-c8bd40b98d43-kube-api-access-zj7lg\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:10.287844 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:10.287810 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:17:10.288276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:10.287860 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 29 22:17:10.316605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:10.316566 4952 generic.go:356] "Generic (PLEG): container finished" podID="a843f495-6b86-44b6-af22-c8bd40b98d43" containerID="6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73" exitCode=137 Oct 29 22:17:10.316798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:10.316630 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9450e4" Oct 29 22:17:10.316798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:10.316667 4952 scope.go:122] "RemoveContainer" containerID="6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73" Oct 29 22:17:10.319833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:10.319807 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-54ff16" event={"ID":"98e4c782-3f72-46ea-97e9-e49f05bc5a37","Type":"ContainerStarted","Data":"9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1"} Oct 29 22:17:10.372962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:10.370067 4952 scope.go:122] "RemoveContainer" containerID="6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73" Oct 29 22:17:10.372962 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:10.370487 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73\": container with ID starting with 6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73 not found: ID does not exist" containerID="6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73" Oct 29 22:17:10.372962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:10.370526 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73"} err="failed to get container status \"6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73\": rpc error: code = NotFound desc = could not find container \"6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73\": container with ID starting with 6e00728f76d4f7d7031bb71c69641de8c14c2ef8afad06599360da1fc9b9bb73 not found: ID does not exist" Oct 29 22:17:10.394607 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:10.394571 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-9450e4\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="a843f495-6b86-44b6-af22-c8bd40b98d43" pod="opendatahub/test-curl-7761-9450e4" Oct 29 22:17:10.677734 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:10.677696 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a843f495-6b86-44b6-af22-c8bd40b98d43" path="/var/lib/kubelet/pods/a843f495-6b86-44b6-af22-c8bd40b98d43/volumes" Oct 29 22:17:11.061142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.053809 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:11.287641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.287608 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:17:11.287798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.287674 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 29 22:17:11.467774 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.466406 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7761-54ff16" podStartSLOduration=3.466385733 podStartE2EDuration="3.466385733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:10.370157635 +0000 UTC m=+2095.844746054" watchObservedRunningTime="2026-10-29 22:17:11.466385733 +0000 UTC m=+2096.940974154" Oct 29 22:17:11.471604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.470534 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-54ff16"] Oct 29 22:17:11.477598 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.477466 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-54ff16"] Oct 29 22:17:11.600687 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.600655 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-8a1c45"] Oct 29 22:17:11.601370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.601354 4952 cpu_manager.go:414] "containerMap: removing container" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" containerName="test-curl-7761-54ff16" Oct 29 22:17:11.601429 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.601372 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" containerName="test-curl-7761-54ff16" Oct 29 22:17:11.601515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.601505 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" containerName="test-curl-7761-54ff16" Oct 29 22:17:11.605150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.605131 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-8a1c45" Oct 29 22:17:11.649808 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.649772 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-8a1c45"] Oct 29 22:17:11.679634 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.675579 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-cd455e" Oct 29 22:17:11.713744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.713700 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61bd515f-80cd-4e6e-95d8-38db04cc350e-kube-api-access-tzjbp\" (UniqueName: \"kubernetes.io/projected/61bd515f-80cd-4e6e-95d8-38db04cc350e-kube-api-access-tzjbp\") pod \"61bd515f-80cd-4e6e-95d8-38db04cc350e\" (UID: \"61bd515f-80cd-4e6e-95d8-38db04cc350e\") " Oct 29 22:17:11.715086 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.715052 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fvd\" (UniqueName: \"kubernetes.io/projected/5b81b774-17c2-48f7-8091-202b805fa03a-kube-api-access-g8fvd\") pod \"test-curl-7761-8a1c45\" (UID: \"5b81b774-17c2-48f7-8091-202b805fa03a\") " pod="opendatahub/test-curl-7761-8a1c45" Oct 29 22:17:11.716602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.716567 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bd515f-80cd-4e6e-95d8-38db04cc350e-kube-api-access-tzjbp" pod "61bd515f-80cd-4e6e-95d8-38db04cc350e" (UID: "61bd515f-80cd-4e6e-95d8-38db04cc350e"). InnerVolumeSpecName "kube-api-access-tzjbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:11.817870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.817836 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fvd\" (UniqueName: \"kubernetes.io/projected/5b81b774-17c2-48f7-8091-202b805fa03a-kube-api-access-g8fvd\") pod \"test-curl-7761-8a1c45\" (UID: \"5b81b774-17c2-48f7-8091-202b805fa03a\") " pod="opendatahub/test-curl-7761-8a1c45" Oct 29 22:17:11.818349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.818209 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzjbp\" (UniqueName: \"kubernetes.io/projected/61bd515f-80cd-4e6e-95d8-38db04cc350e-kube-api-access-tzjbp\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:11.873986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.873383 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fvd\" (UniqueName: \"kubernetes.io/projected/5b81b774-17c2-48f7-8091-202b805fa03a-kube-api-access-g8fvd\") pod \"test-curl-7761-8a1c45\" (UID: \"5b81b774-17c2-48f7-8091-202b805fa03a\") " pod="opendatahub/test-curl-7761-8a1c45" Oct 29 22:17:11.985657 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:11.985623 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-8a1c45" Oct 29 22:17:12.287856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.287768 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:17:12.287856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.287828 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 29 22:17:12.368020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.367294 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-8a1c45"] Oct 29 22:17:12.403361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.403332 4952 generic.go:356] "Generic (PLEG): container finished" podID="61bd515f-80cd-4e6e-95d8-38db04cc350e" containerID="ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25" exitCode=137 Oct 29 22:17:12.403555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.403405 4952 scope.go:122] "RemoveContainer" containerID="ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25" Oct 29 22:17:12.403640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.403431 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-cd455e" Oct 29 22:17:12.404121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.403737 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7761-54ff16" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" containerName="test-curl-7761-54ff16" containerID="cri-o://9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1" gracePeriod=2 Oct 29 22:17:12.417184 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:12.417157 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-54ff16\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:12.440930 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:12.439312 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-54ff16\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:12.500134 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:12.500098 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-54ff16\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:12.500622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.500248 4952 scope.go:122] "RemoveContainer" containerID="ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25" Oct 29 22:17:12.500706 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:12.500662 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25\": container with ID starting with ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25 not found: ID does not exist" containerID="ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25" Oct 29 22:17:12.500706 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.500692 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25"} err="failed to get container status \"ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25\": rpc error: code = NotFound desc = could not find container \"ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25\": container with ID starting with ff114b82a97aaa95891fa99b02f2503b6925b305db9ecdc8fa1c0aa70dce1f25 not found: ID does not exist" Oct 29 22:17:12.513347 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:12.513312 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-cd455e\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="61bd515f-80cd-4e6e-95d8-38db04cc350e" pod="opendatahub/test-curl-7761-cd455e" Oct 29 22:17:12.663792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.663296 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61bd515f-80cd-4e6e-95d8-38db04cc350e" path="/var/lib/kubelet/pods/61bd515f-80cd-4e6e-95d8-38db04cc350e/volumes" Oct 29 22:17:12.966610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:12.966568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:13.287760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:13.287673 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:17:13.287760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:13.287735 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 29 22:17:13.414206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:13.414168 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-8a1c45" event={"ID":"5b81b774-17c2-48f7-8091-202b805fa03a","Type":"ContainerStarted","Data":"5eb5f8cf3d85e0d1b378706b301b553b97e0b2c71f88febdb84ea7c99e27c19b"} Oct 29 22:17:13.414367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:13.414219 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-8a1c45" event={"ID":"5b81b774-17c2-48f7-8091-202b805fa03a","Type":"ContainerStarted","Data":"d58abb00a880c41d9cb5ef1ea97d3f6b8b74963e9e145a19e81c90dde517a918"} Oct 29 22:17:13.459917 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:13.459840 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7761-8a1c45" podStartSLOduration=2.459824151 podStartE2EDuration="2.459824151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:13.453308107 +0000 UTC m=+2098.927896525" watchObservedRunningTime="2026-10-29 22:17:13.459824151 +0000 UTC m=+2098.934412569" Oct 29 22:17:14.288324 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:14.288285 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:17:14.288756 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:14.288350 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 29 22:17:14.385242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:14.385208 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-8a1c45"] Oct 29 22:17:14.390965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:14.390013 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-8a1c45"] Oct 29 22:17:14.777745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:14.777715 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:14.791139 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:14.791100 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-54ff16\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:14.890773 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:14.890006 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:14.994542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:14.994504 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e4c782-3f72-46ea-97e9-e49f05bc5a37-kube-api-access-qm7xq\" (UniqueName: \"kubernetes.io/projected/98e4c782-3f72-46ea-97e9-e49f05bc5a37-kube-api-access-qm7xq\") pod \"98e4c782-3f72-46ea-97e9-e49f05bc5a37\" (UID: \"98e4c782-3f72-46ea-97e9-e49f05bc5a37\") " Oct 29 22:17:14.996777 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:14.996748 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e4c782-3f72-46ea-97e9-e49f05bc5a37-kube-api-access-qm7xq" pod "98e4c782-3f72-46ea-97e9-e49f05bc5a37" (UID: "98e4c782-3f72-46ea-97e9-e49f05bc5a37"). InnerVolumeSpecName "kube-api-access-qm7xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:15.097447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.097409 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm7xq\" (UniqueName: \"kubernetes.io/projected/98e4c782-3f72-46ea-97e9-e49f05bc5a37-kube-api-access-qm7xq\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:15.287615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.287533 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:17:15.287615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.287601 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 29 22:17:15.390554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.390516 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:17:15.394312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.394286 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:17:15.474342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.474304 4952 generic.go:356] "Generic (PLEG): container finished" podID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" containerID="9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1" exitCode=137 Oct 29 22:17:15.474545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.474359 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:15.474545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.474378 4952 scope.go:122] "RemoveContainer" containerID="9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1" Oct 29 22:17:15.474668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.474634 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7761-8a1c45" podUID="5b81b774-17c2-48f7-8091-202b805fa03a" containerName="test-curl-7761-8a1c45" containerID="cri-o://5eb5f8cf3d85e0d1b378706b301b553b97e0b2c71f88febdb84ea7c99e27c19b" gracePeriod=2 Oct 29 22:17:15.492181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.492155 4952 scope.go:122] "RemoveContainer" containerID="9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1" Oct 29 22:17:15.492325 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:15.492247 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-54ff16\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:15.493271 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:15.492698 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1\": container with ID starting with 9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1 not found: ID does not exist" containerID="9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1" Oct 29 22:17:15.493375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:15.493288 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1"} err="failed to get container status \"9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1\": rpc error: code = NotFound desc = could not find container \"9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1\": container with ID starting with 9289596eedcc8f5781885339481ab43b151f673ceee43f0218898c210581dad1 not found: ID does not exist" Oct 29 22:17:15.505982 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:15.505949 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-54ff16\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:15.519407 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:15.519359 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-54ff16\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:15.529635 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:15.529605 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:17:16.160415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:16.160383 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:16.293233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:16.293195 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:16.293745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:16.293724 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:16.294692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:16.294668 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:17:16.306657 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:16.306629 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-54ff16\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:16.379232 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:16.379127 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-54ff16\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" pod="opendatahub/test-curl-7761-54ff16" Oct 29 22:17:16.672090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:16.672043 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e4c782-3f72-46ea-97e9-e49f05bc5a37" path="/var/lib/kubelet/pods/98e4c782-3f72-46ea-97e9-e49f05bc5a37/volumes" Oct 29 22:17:16.684481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:16.684440 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:16.892935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:16.888963 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:17.531650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:17.531605 4952 generic.go:356] "Generic (PLEG): container finished" podID="5b81b774-17c2-48f7-8091-202b805fa03a" containerID="5eb5f8cf3d85e0d1b378706b301b553b97e0b2c71f88febdb84ea7c99e27c19b" exitCode=137 Oct 29 22:17:17.784290 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:17.784259 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-8a1c45" Oct 29 22:17:17.883563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:17.883523 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b81b774-17c2-48f7-8091-202b805fa03a-kube-api-access-g8fvd\" (UniqueName: \"kubernetes.io/projected/5b81b774-17c2-48f7-8091-202b805fa03a-kube-api-access-g8fvd\") pod \"5b81b774-17c2-48f7-8091-202b805fa03a\" (UID: \"5b81b774-17c2-48f7-8091-202b805fa03a\") " Oct 29 22:17:17.885866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:17.885832 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b81b774-17c2-48f7-8091-202b805fa03a-kube-api-access-g8fvd" pod "5b81b774-17c2-48f7-8091-202b805fa03a" (UID: "5b81b774-17c2-48f7-8091-202b805fa03a"). InnerVolumeSpecName "kube-api-access-g8fvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:17.985717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:17.985629 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8fvd\" (UniqueName: \"kubernetes.io/projected/5b81b774-17c2-48f7-8091-202b805fa03a-kube-api-access-g8fvd\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:18.545354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:18.545322 4952 scope.go:122] "RemoveContainer" containerID="5eb5f8cf3d85e0d1b378706b301b553b97e0b2c71f88febdb84ea7c99e27c19b" Oct 29 22:17:18.545542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:18.545486 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-8a1c45" Oct 29 22:17:18.602030 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:18.597079 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7761-8a1c45\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="5b81b774-17c2-48f7-8091-202b805fa03a" pod="opendatahub/test-curl-7761-8a1c45" Oct 29 22:17:18.628794 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:18.628757 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:18.653142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:18.653109 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b81b774-17c2-48f7-8091-202b805fa03a" path="/var/lib/kubelet/pods/5b81b774-17c2-48f7-8091-202b805fa03a/volumes" Oct 29 22:17:20.462929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:20.462047 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:20.788464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:20.788376 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:20.958136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:20.958103 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:22.338943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.337645 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:22.628800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.628719 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:22.658341 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.658306 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz"] Oct 29 22:17:22.659142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.659116 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5b81b774-17c2-48f7-8091-202b805fa03a" containerName="test-curl-7761-8a1c45" Oct 29 22:17:22.659142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.659141 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b81b774-17c2-48f7-8091-202b805fa03a" containerName="test-curl-7761-8a1c45" Oct 29 22:17:22.659351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.659308 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b81b774-17c2-48f7-8091-202b805fa03a" containerName="test-curl-7761-8a1c45" Oct 29 22:17:22.664066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.664042 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:17:22.680520 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.680495 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz"] Oct 29 22:17:22.686610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.686589 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0412c3\"" Oct 29 22:17:22.686610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.686601 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0412c3-dockercfg-kjjld\"" Oct 29 22:17:22.700275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.700234 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dd7134-a80f-4634-946d-1a23575d964b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz\" (UID: \"21dd7134-a80f-4634-946d-1a23575d964b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:17:22.700799 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.700773 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6nj4\" (UniqueName: \"kubernetes.io/projected/21dd7134-a80f-4634-946d-1a23575d964b-kube-api-access-z6nj4\") pod \"payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz\" (UID: \"21dd7134-a80f-4634-946d-1a23575d964b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:17:22.802623 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.802565 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nj4\" (UniqueName: \"kubernetes.io/projected/21dd7134-a80f-4634-946d-1a23575d964b-kube-api-access-z6nj4\") pod \"payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz\" (UID: \"21dd7134-a80f-4634-946d-1a23575d964b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:17:22.802895 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.802872 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dd7134-a80f-4634-946d-1a23575d964b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz\" (UID: \"21dd7134-a80f-4634-946d-1a23575d964b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:17:22.803876 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.803846 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dd7134-a80f-4634-946d-1a23575d964b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz\" (UID: \"21dd7134-a80f-4634-946d-1a23575d964b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:17:22.880976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.862934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:22.950207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:22.950151 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nj4\" (UniqueName: \"kubernetes.io/projected/21dd7134-a80f-4634-946d-1a23575d964b-kube-api-access-z6nj4\") pod \"payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz\" (UID: \"21dd7134-a80f-4634-946d-1a23575d964b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:17:23.280521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:23.280432 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:17:23.614966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:23.614936 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz"] Oct 29 22:17:23.615454 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:17:23.615428 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dd7134_a80f_4634_946d_1a23575d964b.slice/crio-f557ab8caa7293dbd8891327754edf969d61730210a920649b7e6c4f926872c2 WatchSource:0}: Error finding container f557ab8caa7293dbd8891327754edf969d61730210a920649b7e6c4f926872c2: Status 404 returned error can't find the container with id f557ab8caa7293dbd8891327754edf969d61730210a920649b7e6c4f926872c2 Oct 29 22:17:24.218594 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:24.215178 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:24.605645 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:24.605602 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" event={"ID":"21dd7134-a80f-4634-946d-1a23575d964b","Type":"ContainerStarted","Data":"c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0"} Oct 29 22:17:24.605838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:24.605707 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" event={"ID":"21dd7134-a80f-4634-946d-1a23575d964b","Type":"ContainerStarted","Data":"f557ab8caa7293dbd8891327754edf969d61730210a920649b7e6c4f926872c2"} Oct 29 22:17:24.605896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:24.605845 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:17:24.651136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:24.651071 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" podStartSLOduration=2.651050307 podStartE2EDuration="2.651050307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:24.646356148 +0000 UTC m=+2110.120944568" watchObservedRunningTime="2026-10-29 22:17:24.651050307 +0000 UTC m=+2110.125638726" Oct 29 22:17:25.975784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:25.974581 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:27.546193 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.546155 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp"] Oct 29 22:17:27.551338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.551312 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.570288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.570256 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-b7c2d5-serving-cert\"" Oct 29 22:17:27.575958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.575926 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp"] Oct 29 22:17:27.622638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.622607 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.622972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.622948 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-metrics-certs\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.623090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.623026 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qq4\" (UniqueName: \"kubernetes.io/projected/5d367b46-cebf-42bf-93d4-a5c558ec1448-kube-api-access-97qq4\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.623181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.623163 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d367b46-cebf-42bf-93d4-a5c558ec1448-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.684974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.684938 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:27.839848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.839804 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d367b46-cebf-42bf-93d4-a5c558ec1448-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.840338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.840310 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.840507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.840492 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-metrics-certs\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.840612 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.840596 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qq4\" (UniqueName: \"kubernetes.io/projected/5d367b46-cebf-42bf-93d4-a5c558ec1448-kube-api-access-97qq4\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.840677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.840652 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d367b46-cebf-42bf-93d4-a5c558ec1448-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.841275 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:27.841255 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-serving-cert: secret "maas-api-e2e-extmod-a-b7c2d5-serving-cert" not found Oct 29 22:17:27.841428 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:27.841418 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-maas-api-tls podName:5d367b46-cebf-42bf-93d4-a5c558ec1448 nodeName:}" failed. No retries permitted until 2026-10-29 22:17:28.34139862 +0000 UTC m=+2113.815987030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-maas-api-tls") pod "maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" (UID: "5d367b46-cebf-42bf-93d4-a5c558ec1448") : secret "maas-api-e2e-extmod-a-b7c2d5-serving-cert" not found Oct 29 22:17:27.844858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.844832 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-metrics-certs\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:27.853103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.853067 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:27.866157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:27.866126 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qq4\" (UniqueName: \"kubernetes.io/projected/5d367b46-cebf-42bf-93d4-a5c558ec1448-kube-api-access-97qq4\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:28.057787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:28.057747 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:28.349982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:28.349521 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:28.352614 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:28.352584 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:28.471049 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:28.471009 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:28.991389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:28.984391 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp"] Oct 29 22:17:28.991389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:28.988850 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:17:29.657927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:29.657472 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:29.734243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:29.734156 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" event={"ID":"5d367b46-cebf-42bf-93d4-a5c558ec1448","Type":"ContainerStarted","Data":"4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad"} Oct 29 22:17:29.734243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:29.734221 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" event={"ID":"5d367b46-cebf-42bf-93d4-a5c558ec1448","Type":"ContainerStarted","Data":"f5f9acb4e7df0e47b3de7d5d17572d9054ba8ceabaf954c956fb8126c9826a55"} Oct 29 22:17:29.734471 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:29.734262 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:29.779592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:29.779540 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" podStartSLOduration=2.324980289 podStartE2EDuration="2.779526214s" totalImagesPullingTime="454.545925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:29.776583008 +0000 UTC m=+2115.251171428" watchObservedRunningTime="2026-10-29 22:17:29.779526214 +0000 UTC m=+2115.254114631" Oct 29 22:17:31.379350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:31.379312 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:33.172932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:33.171611 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:34.848927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:34.847818 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:35.617788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:35.617754 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:17:35.750444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:35.750412 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:17:36.535359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:36.535314 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:36.819601 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:36.819503 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:38.125303 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:38.125264 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:38.291212 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:38.291171 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:38.779930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:38.779350 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:38.961470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:38.961365 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:39.975143 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:39.975108 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:40.556691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:40.556654 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:17:41.458900 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:41.458863 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl"] Oct 29 22:17:41.459523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:41.459246 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" containerID="cri-o://cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27" gracePeriod=30 Oct 29 22:17:41.844885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:41.844846 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:42.098153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:42.098057 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:42.286562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:42.286525 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:43.863833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:43.863799 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:44.036759 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:44.036719 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:45.551066 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:45.551022 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:17:45.603918 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:45.603871 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:45.888138 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:45.888049 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:46.067653 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.067620 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:46.857972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.857941 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 22:17:46.926055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.926016 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-certs\") pod \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " Oct 29 22:17:46.926260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.926156 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-podinfo\") pod \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " Oct 29 22:17:46.926260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.926221 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-data\") pod \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " Oct 29 22:17:46.926260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.926247 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-socket\") pod \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " Oct 29 22:17:46.926506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.926284 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-token\" (UniqueName: \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-token\") pod \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " Oct 29 22:17:46.926506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.926368 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-envoy\") pod \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " Oct 29 22:17:46.926744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.926518 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-credential-socket\") pod \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " Oct 29 22:17:46.926744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.926606 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-kube-api-access-kf7nt\" (UniqueName: \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-kube-api-access-kf7nt\") pod \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " Oct 29 22:17:46.926868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.926764 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istiod-ca-cert\") pod \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\" (UID: \"78936c9a-79a4-48f5-b4f1-a2087bd3d2da\") " Oct 29 22:17:46.927308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.927260 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-socket" pod "78936c9a-79a4-48f5-b4f1-a2087bd3d2da" (UID: "78936c9a-79a4-48f5-b4f1-a2087bd3d2da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:46.927526 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.927502 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-certs" pod "78936c9a-79a4-48f5-b4f1-a2087bd3d2da" (UID: "78936c9a-79a4-48f5-b4f1-a2087bd3d2da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:46.927846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.927818 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-credential-socket" pod "78936c9a-79a4-48f5-b4f1-a2087bd3d2da" (UID: "78936c9a-79a4-48f5-b4f1-a2087bd3d2da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:46.929232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.928836 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:46.929232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.928865 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:46.929232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.928882 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:46.929232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.929202 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-data" pod "78936c9a-79a4-48f5-b4f1-a2087bd3d2da" (UID: "78936c9a-79a4-48f5-b4f1-a2087bd3d2da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:46.930437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.930381 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istiod-ca-cert" pod "78936c9a-79a4-48f5-b4f1-a2087bd3d2da" (UID: "78936c9a-79a4-48f5-b4f1-a2087bd3d2da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:17:46.931028 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.930992 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-podinfo" pod "78936c9a-79a4-48f5-b4f1-a2087bd3d2da" (UID: "78936c9a-79a4-48f5-b4f1-a2087bd3d2da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:17:46.933704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.933678 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-token" pod "78936c9a-79a4-48f5-b4f1-a2087bd3d2da" (UID: "78936c9a-79a4-48f5-b4f1-a2087bd3d2da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:46.937325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.937289 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-envoy" pod "78936c9a-79a4-48f5-b4f1-a2087bd3d2da" (UID: "78936c9a-79a4-48f5-b4f1-a2087bd3d2da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:17:46.937464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.937330 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-kube-api-access-kf7nt" pod "78936c9a-79a4-48f5-b4f1-a2087bd3d2da" (UID: "78936c9a-79a4-48f5-b4f1-a2087bd3d2da"). InnerVolumeSpecName "kube-api-access-kf7nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:17:46.999665 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.999625 4952 generic.go:356] "Generic (PLEG): container finished" podID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerID="cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27" exitCode=0 Oct 29 22:17:46.999858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.999740 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" event={"ID":"78936c9a-79a4-48f5-b4f1-a2087bd3d2da","Type":"ContainerDied","Data":"cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27"} Oct 29 22:17:46.999858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.999790 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" event={"ID":"78936c9a-79a4-48f5-b4f1-a2087bd3d2da","Type":"ContainerDied","Data":"57e871e407126e34cc659091371283883894bbb3eee66dc304ebcd10656c724e"} Oct 29 22:17:46.999858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.999819 4952 scope.go:122] "RemoveContainer" containerID="cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27" Oct 29 22:17:47.000072 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:46.999986 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" Oct 29 22:17:47.028814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.028792 4952 scope.go:122] "RemoveContainer" containerID="cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27" Oct 29 22:17:47.029162 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:17:47.029139 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27\": container with ID starting with cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27 not found: ID does not exist" containerID="cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27" Oct 29 22:17:47.029324 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.029294 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27"} err="failed to get container status \"cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27\": rpc error: code = NotFound desc = could not find container \"cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27\": container with ID starting with cea1960df2605ad19c8daeb21a1904ee89d711a5c2f71e62eb2253106ea70d27 not found: ID does not exist" Oct 29 22:17:47.030037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.030017 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:47.030121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.030039 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:47.030121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.030049 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:47.030121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.030057 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:47.030121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.030065 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7nt\" (UniqueName: \"kubernetes.io/projected/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-kube-api-access-kf7nt\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:47.030121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.030075 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78936c9a-79a4-48f5-b4f1-a2087bd3d2da-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:17:47.077778 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.077745 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl"] Oct 29 22:17:47.083696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.083658 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl"] Oct 29 22:17:47.493987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.489237 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:47.659781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.659742 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-lvmzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:17:47.659958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:47.659804 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-lvmzl" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:17:48.648964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:48.648928 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" path="/var/lib/kubelet/pods/78936c9a-79a4-48f5-b4f1-a2087bd3d2da/volumes" Oct 29 22:17:49.135925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:49.135369 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:50.733196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:50.733158 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:52.334876 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:52.334833 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:53.010979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.010935 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r"] Oct 29 22:17:53.011662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.011647 4952 cpu_manager.go:414] "containerMap: removing container" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" Oct 29 22:17:53.011730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.011663 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" Oct 29 22:17:53.011797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.011787 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="78936c9a-79a4-48f5-b4f1-a2087bd3d2da" containerName="istio-proxy" Oct 29 22:17:53.016918 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.016889 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.040826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.040793 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r"] Oct 29 22:17:53.043081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.043055 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.043475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.043102 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.043475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.043222 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.043475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.043312 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.043475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.043415 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73907711-dc9d-438f-b43f-a0385e6402be-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.043475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.043470 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.043800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.043572 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73907711-dc9d-438f-b43f-a0385e6402be-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.043800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.043609 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgncb\" (UniqueName: \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-kube-api-access-zgncb\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.043800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.043644 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.147888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.146589 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73907711-dc9d-438f-b43f-a0385e6402be-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.147888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.146647 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.147888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.146787 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73907711-dc9d-438f-b43f-a0385e6402be-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.147888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.146835 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgncb\" (UniqueName: \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-kube-api-access-zgncb\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.147888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.146885 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.147888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.147015 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.147888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.147066 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.147888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.147123 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.147888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.147225 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.149581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.149002 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-data\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.149581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.149002 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-certs\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.149581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.149380 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.150042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.149981 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73907711-dc9d-438f-b43f-a0385e6402be-istiod-ca-cert\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.154426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.150397 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-credential-socket\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.166681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.166649 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-envoy\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.168076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.168047 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73907711-dc9d-438f-b43f-a0385e6402be-istio-podinfo\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.173776 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.173753 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-istio-token\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.179469 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.179445 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgncb\" (UniqueName: \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-kube-api-access-zgncb\") pod \"e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.332702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.332664 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:53.683252 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:17:53.683219 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73907711_dc9d_438f_b43f_a0385e6402be.slice/crio-779050673f5e238ac9fbfbe7bf2c1cc6679ade32db78b85dec431378fb8c9e95 WatchSource:0}: Error finding container 779050673f5e238ac9fbfbe7bf2c1cc6679ade32db78b85dec431378fb8c9e95: Status 404 returned error can't find the container with id 779050673f5e238ac9fbfbe7bf2c1cc6679ade32db78b85dec431378fb8c9e95 Oct 29 22:17:53.683680 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.683587 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r"] Oct 29 22:17:53.686316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.686285 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:17:53.686405 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.686348 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:17:53.686405 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:53.686375 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:17:54.037371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:54.037209 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:54.098966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:54.098864 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" event={"ID":"73907711-dc9d-438f-b43f-a0385e6402be","Type":"ContainerStarted","Data":"8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e"} Oct 29 22:17:54.098966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:54.098965 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" event={"ID":"73907711-dc9d-438f-b43f-a0385e6402be","Type":"ContainerStarted","Data":"779050673f5e238ac9fbfbe7bf2c1cc6679ade32db78b85dec431378fb8c9e95"} Oct 29 22:17:54.152959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:54.150706 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" podStartSLOduration=2.150686007 podStartE2EDuration="2.150686007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:54.145242016 +0000 UTC m=+2139.619830436" watchObservedRunningTime="2026-10-29 22:17:54.150686007 +0000 UTC m=+2139.625274426" Oct 29 22:17:54.333115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:54.333076 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:17:54.334557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:54.334529 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:17:54.334706 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:54.334579 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 22:17:55.338931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:55.336011 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:17:55.338931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:55.336077 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 22:17:55.752469 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:55.752374 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:56.348612 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:56.346846 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:17:56.348612 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:56.346936 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 22:17:57.365931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.334294 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:17:57.365931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.334356 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 22:17:57.379533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.375708 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p"] Oct 29 22:17:57.385161 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.385135 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.388047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.386757 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p"] Oct 29 22:17:57.406480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.405270 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0412c3-serving-cert\"" Oct 29 22:17:57.492376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.490466 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-metrics-certs\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.492376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.490703 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26bk7\" (UniqueName: \"kubernetes.io/projected/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-kube-api-access-26bk7\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.492376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.490807 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.492376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.490987 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.555959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.555897 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:57.596931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.596713 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.596931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.596924 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.597203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.597010 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-metrics-certs\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.597203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.597164 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26bk7\" (UniqueName: \"kubernetes.io/projected/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-kube-api-access-26bk7\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.612957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.598072 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.678717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.677672 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-metrics-certs\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.713953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.713000 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26bk7\" (UniqueName: \"kubernetes.io/projected/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-kube-api-access-26bk7\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.728938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.716757 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.728938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.724253 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:57.784814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:57.784769 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:58.184875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:58.184843 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p"] Oct 29 22:17:58.189006 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:17:58.188973 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b1d6bd_d0af_4db4_9d87_e50a8116a21e.slice/crio-38aebf99ebd88b2d4fcfeca623686ee98ef7286829bcfd889054cb7fbe774246 WatchSource:0}: Error finding container 38aebf99ebd88b2d4fcfeca623686ee98ef7286829bcfd889054cb7fbe774246: Status 404 returned error can't find the container with id 38aebf99ebd88b2d4fcfeca623686ee98ef7286829bcfd889054cb7fbe774246 Oct 29 22:17:58.335414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:58.335370 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:17:58.338972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:58.338936 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 22:17:59.200119 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:59.199999 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" event={"ID":"59b1d6bd-d0af-4db4-9d87-e50a8116a21e","Type":"ContainerStarted","Data":"1e6f1592559a66bea57b862ac7dc602962878dc7f4e8757ef25294fb67ffd68e"} Oct 29 22:17:59.200119 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:59.200078 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" event={"ID":"59b1d6bd-d0af-4db4-9d87-e50a8116a21e","Type":"ContainerStarted","Data":"38aebf99ebd88b2d4fcfeca623686ee98ef7286829bcfd889054cb7fbe774246"} Oct 29 22:17:59.200671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:59.200281 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:17:59.257007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:59.256933 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" podStartSLOduration=1.5730280620000001 podStartE2EDuration="2.256897315s" totalImagesPullingTime="683.869253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:59.254479473 +0000 UTC m=+2144.729067892" watchObservedRunningTime="2026-10-29 22:17:59.256897315 +0000 UTC m=+2144.731485732" Oct 29 22:17:59.344931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:59.335078 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:17:59.344931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:59.335140 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 22:17:59.457049 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:17:59.456955 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:00.333626 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:00.333586 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 29 22:18:00.334196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:00.333650 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 29 22:18:01.154191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:01.154158 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:01.340188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:01.340153 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:18:01.340736 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:01.340711 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:18:01.342480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:01.342454 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:18:02.814948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:02.814879 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:04.503617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:04.503578 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:05.215208 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:05.215176 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:18:06.213097 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:06.213056 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:07.906738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:07.906697 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:09.724632 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:09.724589 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:09.891791 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:09.891752 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:11.379873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:11.377603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:11.689734 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:11.689615 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:13.152936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:13.151459 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:13.378988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:13.377155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:14.199945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:14.172286 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:14.462542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:14.459583 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:14.810424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:14.810302 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:14.977690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:14.977598 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:15.163149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:15.163110 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:15.544872 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:18:15.544773 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:18:17.041714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:17.041663 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:17.570056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:17.570019 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:17.744522 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:17.744420 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:19.048692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:19.048648 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:19.473328 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:19.473289 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:19.646483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:19.646446 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:20.771053 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:20.771005 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:20.943327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:20.943148 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:22.587621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:22.587574 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:24.321848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:24.321794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:25.961930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:25.958163 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:27.579105 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:27.579058 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:29.252550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:29.252494 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:30.926926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:30.923300 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:32.598555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:32.598520 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:34.169952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:34.169874 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:35.813180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:35.813132 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:37.433926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:37.423642 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:39.013004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:39.012958 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:40.840463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:40.833096 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:42.553340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:42.553298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:42.960350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:42.957572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:44.197069 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:44.197031 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:44.412529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:44.412493 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:44.863489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:44.863443 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:45.119201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:45.119105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:45.533009 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:18:45.532913 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:18:46.026531 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:46.021702 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:47.829937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:47.829877 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:49.009132 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:49.009094 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:49.192149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:49.192098 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:49.581935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:49.571325 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:50.746645 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:50.746603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:50.919835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:50.919799 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:51.638535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:51.638482 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:53.374734 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:53.374688 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:55.030927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:55.024645 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:56.544978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:56.544106 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:56.713368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:56.713330 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:56.885618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:56.885544 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:58.375028 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:18:58.374983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:00.034438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:00.034401 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:01.382118 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:01.382078 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:19:02.018360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:02.018324 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:04.207592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:04.207556 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:06.064080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:06.064038 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:07.707000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:07.704833 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:09.313026 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:09.312984 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:09.479458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:09.479418 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:09.975880 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:09.975841 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:10.207981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:10.207943 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:10.728549 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:10.728427 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:11.028819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:11.028731 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:11.216303 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:11.216269 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:11.401931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:11.401431 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:12.684091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:12.684054 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:14.446833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:14.446796 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:15.255236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:15.255200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:15.435950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:15.435871 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:15.535703 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:19:15.535610 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:19:16.288834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:16.288775 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:17.023513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:17.023465 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:17.217540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:17.217406 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:18.241979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:18.241941 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:19.912441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:19.908132 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:21.516994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:21.513896 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:22.757278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:22.757243 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc"] Oct 29 22:19:22.757783 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:22.757443 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" containerID="cri-o://77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87" gracePeriod=30 Oct 29 22:19:23.230929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:23.227633 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:24.915280 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:24.915237 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:26.524932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:26.523557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:28.081251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.081224 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:19:28.180312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.180275 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-envoy\") pod \"938f28fc-c391-46db-aa26-cb709c6eb9ec\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " Oct 29 22:19:28.180510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.180422 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-token\" (UniqueName: \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-token\") pod \"938f28fc-c391-46db-aa26-cb709c6eb9ec\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " Oct 29 22:19:28.180510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.180474 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-kube-api-access-vzzlz\" (UniqueName: \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-kube-api-access-vzzlz\") pod \"938f28fc-c391-46db-aa26-cb709c6eb9ec\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " Oct 29 22:19:28.181287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.180519 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-podinfo\") pod \"938f28fc-c391-46db-aa26-cb709c6eb9ec\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " Oct 29 22:19:28.181287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.180561 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-credential-socket\") pod \"938f28fc-c391-46db-aa26-cb709c6eb9ec\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " Oct 29 22:19:28.181287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.180747 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938f28fc-c391-46db-aa26-cb709c6eb9ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/938f28fc-c391-46db-aa26-cb709c6eb9ec-istiod-ca-cert\") pod \"938f28fc-c391-46db-aa26-cb709c6eb9ec\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " Oct 29 22:19:28.181287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.180841 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-data\") pod \"938f28fc-c391-46db-aa26-cb709c6eb9ec\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " Oct 29 22:19:28.181287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181005 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-certs\") pod \"938f28fc-c391-46db-aa26-cb709c6eb9ec\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " Oct 29 22:19:28.181287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181053 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-socket\") pod \"938f28fc-c391-46db-aa26-cb709c6eb9ec\" (UID: \"938f28fc-c391-46db-aa26-cb709c6eb9ec\") " Oct 29 22:19:28.181287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181221 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-data" pod "938f28fc-c391-46db-aa26-cb709c6eb9ec" (UID: "938f28fc-c391-46db-aa26-cb709c6eb9ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.181661 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181340 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-certs" pod "938f28fc-c391-46db-aa26-cb709c6eb9ec" (UID: "938f28fc-c391-46db-aa26-cb709c6eb9ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.181661 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181520 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-socket" pod "938f28fc-c391-46db-aa26-cb709c6eb9ec" (UID: "938f28fc-c391-46db-aa26-cb709c6eb9ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.181661 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181567 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-credential-socket" pod "938f28fc-c391-46db-aa26-cb709c6eb9ec" (UID: "938f28fc-c391-46db-aa26-cb709c6eb9ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.181817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181730 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938f28fc-c391-46db-aa26-cb709c6eb9ec-istiod-ca-cert" pod "938f28fc-c391-46db-aa26-cb709c6eb9ec" (UID: "938f28fc-c391-46db-aa26-cb709c6eb9ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:28.181956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181931 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.182068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181958 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.182068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181973 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.182068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.181988 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/938f28fc-c391-46db-aa26-cb709c6eb9ec-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.182068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.182002 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.184792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.184763 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-envoy" pod "938f28fc-c391-46db-aa26-cb709c6eb9ec" (UID: "938f28fc-c391-46db-aa26-cb709c6eb9ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:28.185009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.184981 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-token" pod "938f28fc-c391-46db-aa26-cb709c6eb9ec" (UID: "938f28fc-c391-46db-aa26-cb709c6eb9ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:28.185267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.185244 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-podinfo" pod "938f28fc-c391-46db-aa26-cb709c6eb9ec" (UID: "938f28fc-c391-46db-aa26-cb709c6eb9ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:28.185357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.185285 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-kube-api-access-vzzlz" pod "938f28fc-c391-46db-aa26-cb709c6eb9ec" (UID: "938f28fc-c391-46db-aa26-cb709c6eb9ec"). InnerVolumeSpecName "kube-api-access-vzzlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:28.191156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.191125 4952 generic.go:356] "Generic (PLEG): container finished" podID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerID="77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87" exitCode=0 Oct 29 22:19:28.191282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.191196 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" Oct 29 22:19:28.191860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.191380 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" event={"ID":"938f28fc-c391-46db-aa26-cb709c6eb9ec","Type":"ContainerDied","Data":"77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87"} Oct 29 22:19:28.191860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.191449 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc" event={"ID":"938f28fc-c391-46db-aa26-cb709c6eb9ec","Type":"ContainerDied","Data":"3a34c96c843c4eab818c7f0d5e8031e848781f34a5c22b9dc3647f94d309352e"} Oct 29 22:19:28.191860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.191486 4952 scope.go:122] "RemoveContainer" containerID="77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87" Oct 29 22:19:28.225105 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.225081 4952 scope.go:122] "RemoveContainer" containerID="77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87" Oct 29 22:19:28.225452 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:19:28.225431 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87\": container with ID starting with 77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87 not found: ID does not exist" containerID="77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87" Oct 29 22:19:28.225518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.225462 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87"} err="failed to get container status \"77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87\": rpc error: code = NotFound desc = could not find container \"77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87\": container with ID starting with 77006741986c793382d8e6bdb3b2fc942d0eda9ce7f531c3838a39833e389c87 not found: ID does not exist" Oct 29 22:19:28.262136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.262110 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc"] Oct 29 22:19:28.269409 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.269384 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-8sdwc"] Oct 29 22:19:28.283611 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.283590 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.283714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.283614 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.283714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.283624 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzzlz\" (UniqueName: \"kubernetes.io/projected/938f28fc-c391-46db-aa26-cb709c6eb9ec-kube-api-access-vzzlz\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.283714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.283636 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/938f28fc-c391-46db-aa26-cb709c6eb9ec-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:28.369927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.369021 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:28.649042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.648944 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" path="/var/lib/kubelet/pods/938f28fc-c391-46db-aa26-cb709c6eb9ec/volumes" Oct 29 22:19:28.987595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.987516 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4"] Oct 29 22:19:28.987749 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:28.987720 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" containerID="cri-o://9da487476506349653425611044431bb95a9622a77bffb7c018b53b17c84a814" gracePeriod=30 Oct 29 22:19:29.989923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:29.989872 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:30.173746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:30.173700 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:30.340126 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:30.340087 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:31.243578 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.243538 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7"] Oct 29 22:19:31.244435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.244415 4952 cpu_manager.go:414] "containerMap: removing container" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" Oct 29 22:19:31.244502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.244436 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" Oct 29 22:19:31.244603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.244587 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="938f28fc-c391-46db-aa26-cb709c6eb9ec" containerName="istio-proxy" Oct 29 22:19:31.250047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.250025 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.280837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.280808 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-bef83b-openshift-default-dockercfg-hrrzn\"" Oct 29 22:19:31.288587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.288555 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7"] Oct 29 22:19:31.370345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.370310 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d700972-dd64-4e07-b92f-27fb31c21484-istiod-ca-cert\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.370345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.370350 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.370615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.370375 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-istio-token\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.370679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.370616 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-data\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.370736 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.370710 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-credential-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.370850 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.370835 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d700972-dd64-4e07-b92f-27fb31c21484-istio-podinfo\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.370934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.370874 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbvlt\" (UniqueName: \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-kube-api-access-qbvlt\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.371050 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.371035 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-envoy\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.371114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.371088 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-certs\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.472391 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.472356 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d700972-dd64-4e07-b92f-27fb31c21484-istio-podinfo\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.472561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.472397 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbvlt\" (UniqueName: \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-kube-api-access-qbvlt\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.472561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.472487 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-envoy\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.472561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.472533 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-certs\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.472714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.472592 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d700972-dd64-4e07-b92f-27fb31c21484-istiod-ca-cert\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.472714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.472619 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.472714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.472653 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-istio-token\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.472887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.472791 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-data\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.473034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.473007 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-credential-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.473157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.473123 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.473157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.473142 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-certs\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.473270 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.473206 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-data\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.473402 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.473385 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-credential-socket\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.473483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.473457 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d700972-dd64-4e07-b92f-27fb31c21484-istiod-ca-cert\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.475011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.474986 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-envoy\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.475328 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.475308 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d700972-dd64-4e07-b92f-27fb31c21484-istio-podinfo\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.497614 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.497555 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbvlt\" (UniqueName: \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-kube-api-access-qbvlt\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.497786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.497761 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-istio-token\") pod \"e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.565859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.565820 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:31.936443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.935547 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:31.975385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.970204 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7"] Oct 29 22:19:31.980102 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:19:31.980059 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d700972_dd64_4e07_b92f_27fb31c21484.slice/crio-a8404671e2867828dfb6386243049d7270075e72e146b19f396c36b72c695d12 WatchSource:0}: Error finding container a8404671e2867828dfb6386243049d7270075e72e146b19f396c36b72c695d12: Status 404 returned error can't find the container with id a8404671e2867828dfb6386243049d7270075e72e146b19f396c36b72c695d12 Oct 29 22:19:31.997024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.996985 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:19:31.997180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.997070 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:19:31.997180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:31.997117 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:19:32.243981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:32.240839 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" event={"ID":"4d700972-dd64-4e07-b92f-27fb31c21484","Type":"ContainerStarted","Data":"1a0f0b2000f8f89c50c528196601bd0b906e0a5355278ecf058dfb7bf8f76311"} Oct 29 22:19:32.243981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:32.240939 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" event={"ID":"4d700972-dd64-4e07-b92f-27fb31c21484","Type":"ContainerStarted","Data":"a8404671e2867828dfb6386243049d7270075e72e146b19f396c36b72c695d12"} Oct 29 22:19:32.295716 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:32.295654 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" podStartSLOduration=1.2956392829999999 podStartE2EDuration="1.295639283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:32.288143676 +0000 UTC m=+2237.762732094" watchObservedRunningTime="2026-10-29 22:19:32.295639283 +0000 UTC m=+2237.770227764" Oct 29 22:19:32.566280 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:32.566189 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:33.568764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:33.568727 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:19:33.569254 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:33.568788 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 29 22:19:33.756529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:33.756493 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:34.311948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.311840 4952 generic.go:356] "Generic (PLEG): container finished" podID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerID="9da487476506349653425611044431bb95a9622a77bffb7c018b53b17c84a814" exitCode=0 Oct 29 22:19:34.312479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.312432 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" event={"ID":"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446","Type":"ContainerDied","Data":"9da487476506349653425611044431bb95a9622a77bffb7c018b53b17c84a814"} Oct 29 22:19:34.492968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.492932 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:19:34.569975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.566716 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:19:34.569975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.566786 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 29 22:19:34.602196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602165 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-token\" (UniqueName: \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-token\") pod \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " Oct 29 22:19:34.602375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602258 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-certs\") pod \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " Oct 29 22:19:34.602375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602297 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-envoy\") pod \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " Oct 29 22:19:34.602375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602330 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-data\") pod \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " Oct 29 22:19:34.602567 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602473 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istiod-ca-cert\") pod \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " Oct 29 22:19:34.602567 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602519 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-credential-socket\") pod \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " Oct 29 22:19:34.602688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602596 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-certs" pod "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" (UID: "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:34.602743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602687 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-podinfo\") pod \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " Oct 29 22:19:34.602743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602733 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-credential-socket" pod "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" (UID: "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:34.602840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602785 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istiod-ca-cert" pod "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" (UID: "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:19:34.602952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602934 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-kube-api-access-xl7mg\" (UniqueName: \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-kube-api-access-xl7mg\") pod \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " Oct 29 22:19:34.603023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.602971 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-socket\") pod \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\" (UID: \"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446\") " Oct 29 22:19:34.603305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.603273 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-socket" pod "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" (UID: "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:34.603393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.603364 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-data" pod "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" (UID: "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:34.603727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.603708 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:34.603805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.603731 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:34.603805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.603745 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:34.603805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.603758 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:34.603805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.603773 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:34.605233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.605198 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-envoy" pod "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" (UID: "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:19:34.605352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.605331 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-token" pod "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" (UID: "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:34.605399 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.605360 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-podinfo" pod "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" (UID: "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:19:34.605591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.605572 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-kube-api-access-xl7mg" pod "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" (UID: "6dcaf3ea-6b6e-4d3d-865d-55987e5f9446"). InnerVolumeSpecName "kube-api-access-xl7mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:19:34.704814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.704786 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:34.704814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.704815 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7mg\" (UniqueName: \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-kube-api-access-xl7mg\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:34.705033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.704826 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:34.705033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:34.704835 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:19:35.336365 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:35.336324 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" event={"ID":"6dcaf3ea-6b6e-4d3d-865d-55987e5f9446","Type":"ContainerDied","Data":"26e177c868d29b2ebea1b00202ac5cbc57e3867696575165cc26d30071c2e917"} Oct 29 22:19:35.336650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:35.336622 4952 scope.go:122] "RemoveContainer" containerID="9da487476506349653425611044431bb95a9622a77bffb7c018b53b17c84a814" Oct 29 22:19:35.339243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:35.339226 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4" Oct 29 22:19:35.457833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:35.456503 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:35.473593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:35.471959 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4"] Oct 29 22:19:35.510938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:35.499851 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-gk9f4"] Oct 29 22:19:35.569351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:35.569315 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:19:35.569533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:35.569385 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 29 22:19:36.566834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:36.566799 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:19:36.567357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:36.566851 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 29 22:19:36.666044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:36.665996 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" path="/var/lib/kubelet/pods/6dcaf3ea-6b6e-4d3d-865d-55987e5f9446/volumes" Oct 29 22:19:37.275930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:37.272233 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:37.567438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:37.567349 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:19:37.567438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:37.567423 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 29 22:19:38.566414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:38.566371 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 29 22:19:38.566857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:38.566802 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 29 22:19:39.583936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:39.580222 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:39.583936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:39.581945 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:39.589514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:39.589485 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:19:39.753511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:39.753469 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:40.073022 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:40.072930 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:40.868398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:40.868357 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:41.551628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:41.551586 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:43.563499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:43.563255 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:44.666995 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:44.666862 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:45.259966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:45.256328 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:45.532291 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:19:45.532202 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:19:45.837958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:45.837872 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:46.005532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:46.005498 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:46.189554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:46.189463 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:46.491574 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:46.491468 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:46.669170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:46.669137 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:47.294356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:47.294317 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:47.599597 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:47.599557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:47.877488 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:47.877363 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:49.295534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:49.295501 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:49.982928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:49.959577 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:51.355826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:51.355786 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:52.912713 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:52.912663 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:54.553438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:54.553387 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:56.063935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:56.060894 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:57.734000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:57.732778 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:59.549163 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:59.549121 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:59.731924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:19:59.731872 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:01.284535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:01.284494 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:02.926337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:02.926287 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:04.530927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:04.524770 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:06.115805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:06.115765 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n"] Oct 29 22:20:06.208947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:06.207811 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:06.377565 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:06.377465 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:07.841547 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:07.841514 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:09.643952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:09.643890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:10.107617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:10.107576 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:10.322307 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:10.322272 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:11.264926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:11.264420 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:12.884336 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:12.884267 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:12.968107 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:12.968071 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp"] Oct 29 22:20:12.968331 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:12.968304 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" containerID="cri-o://3c7bd49f2ec0cf5d178a70a9a867a6bf91b041c7acda8b3c666918d7e9fb6e94" gracePeriod=30 Oct 29 22:20:14.478596 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:14.478557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:15.530174 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:20:15.530139 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:20:16.159523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:16.159309 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:18.040066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.040027 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:18.220031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.219991 4952 generic.go:356] "Generic (PLEG): container finished" podID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerID="3c7bd49f2ec0cf5d178a70a9a867a6bf91b041c7acda8b3c666918d7e9fb6e94" exitCode=0 Oct 29 22:20:18.220224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.220070 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" event={"ID":"55eb0002-813c-4f3f-9951-4bbe18a8802c","Type":"ContainerDied","Data":"3c7bd49f2ec0cf5d178a70a9a867a6bf91b041c7acda8b3c666918d7e9fb6e94"} Oct 29 22:20:18.302091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.302066 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:20:18.468753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.468716 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:18.483179 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483140 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-socket\") pod \"55eb0002-813c-4f3f-9951-4bbe18a8802c\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " Oct 29 22:20:18.483367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483270 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-podinfo\") pod \"55eb0002-813c-4f3f-9951-4bbe18a8802c\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " Oct 29 22:20:18.483367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483304 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-certs\") pod \"55eb0002-813c-4f3f-9951-4bbe18a8802c\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " Oct 29 22:20:18.483489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483390 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55eb0002-813c-4f3f-9951-4bbe18a8802c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55eb0002-813c-4f3f-9951-4bbe18a8802c-istiod-ca-cert\") pod \"55eb0002-813c-4f3f-9951-4bbe18a8802c\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " Oct 29 22:20:18.483489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483449 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-socket" pod "55eb0002-813c-4f3f-9951-4bbe18a8802c" (UID: "55eb0002-813c-4f3f-9951-4bbe18a8802c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.483619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483502 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-credential-socket\") pod \"55eb0002-813c-4f3f-9951-4bbe18a8802c\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " Oct 29 22:20:18.483692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483641 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-certs" pod "55eb0002-813c-4f3f-9951-4bbe18a8802c" (UID: "55eb0002-813c-4f3f-9951-4bbe18a8802c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.483752 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483708 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-envoy\") pod \"55eb0002-813c-4f3f-9951-4bbe18a8802c\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " Oct 29 22:20:18.483893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483873 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-kube-api-access-qrp67\" (UniqueName: \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-kube-api-access-qrp67\") pod \"55eb0002-813c-4f3f-9951-4bbe18a8802c\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " Oct 29 22:20:18.483984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483955 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-token\" (UniqueName: \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-token\") pod \"55eb0002-813c-4f3f-9951-4bbe18a8802c\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " Oct 29 22:20:18.484048 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.483988 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-data\") pod \"55eb0002-813c-4f3f-9951-4bbe18a8802c\" (UID: \"55eb0002-813c-4f3f-9951-4bbe18a8802c\") " Oct 29 22:20:18.484865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.484838 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.485002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.484866 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.485222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.485197 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55eb0002-813c-4f3f-9951-4bbe18a8802c-istiod-ca-cert" pod "55eb0002-813c-4f3f-9951-4bbe18a8802c" (UID: "55eb0002-813c-4f3f-9951-4bbe18a8802c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:20:18.485821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.485794 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-data" pod "55eb0002-813c-4f3f-9951-4bbe18a8802c" (UID: "55eb0002-813c-4f3f-9951-4bbe18a8802c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.485821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.485805 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-podinfo" pod "55eb0002-813c-4f3f-9951-4bbe18a8802c" (UID: "55eb0002-813c-4f3f-9951-4bbe18a8802c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:20:18.486040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.485938 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-credential-socket" pod "55eb0002-813c-4f3f-9951-4bbe18a8802c" (UID: "55eb0002-813c-4f3f-9951-4bbe18a8802c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.487699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.487674 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-kube-api-access-qrp67" pod "55eb0002-813c-4f3f-9951-4bbe18a8802c" (UID: "55eb0002-813c-4f3f-9951-4bbe18a8802c"). InnerVolumeSpecName "kube-api-access-qrp67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:18.487946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.487899 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-token" pod "55eb0002-813c-4f3f-9951-4bbe18a8802c" (UID: "55eb0002-813c-4f3f-9951-4bbe18a8802c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:18.488594 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.488567 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-envoy" pod "55eb0002-813c-4f3f-9951-4bbe18a8802c" (UID: "55eb0002-813c-4f3f-9951-4bbe18a8802c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:18.604939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.603998 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.604939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.604043 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55eb0002-813c-4f3f-9951-4bbe18a8802c-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.604939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.604062 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.604939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.604076 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.604939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.604091 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrp67\" (UniqueName: \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-kube-api-access-qrp67\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.604939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.604110 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:18.604939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:18.604124 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55eb0002-813c-4f3f-9951-4bbe18a8802c-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:19.230834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:19.230803 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" Oct 29 22:20:19.231383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:19.230831 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp" event={"ID":"55eb0002-813c-4f3f-9951-4bbe18a8802c","Type":"ContainerDied","Data":"646cc1b0939622087444675a43b2e5132e76acf2fd43fc05802aa6d21d3901f3"} Oct 29 22:20:19.231383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:19.230899 4952 scope.go:122] "RemoveContainer" containerID="3c7bd49f2ec0cf5d178a70a9a867a6bf91b041c7acda8b3c666918d7e9fb6e94" Oct 29 22:20:19.316237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:19.316202 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp"] Oct 29 22:20:19.328012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:19.327970 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-fcrwp"] Oct 29 22:20:20.054253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:20.054210 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:20.651090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:20.651053 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" path="/var/lib/kubelet/pods/55eb0002-813c-4f3f-9951-4bbe18a8802c/volumes" Oct 29 22:20:21.144361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.144293 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" podUID="cdaaa603-46cc-43ed-a102-9005c08df63f" containerName="main" containerID="cri-o://68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292" gracePeriod=45 Oct 29 22:20:21.240945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.240887 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:21.479726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.479698 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:20:21.578480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.578449 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdaaa603-46cc-43ed-a102-9005c08df63f-kube-api-access-vqshk\" (UniqueName: \"kubernetes.io/projected/cdaaa603-46cc-43ed-a102-9005c08df63f-kube-api-access-vqshk\") pod \"cdaaa603-46cc-43ed-a102-9005c08df63f\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " Oct 29 22:20:21.578480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.578491 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-dshm\") pod \"cdaaa603-46cc-43ed-a102-9005c08df63f\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " Oct 29 22:20:21.578836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.578569 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-model-cache\") pod \"cdaaa603-46cc-43ed-a102-9005c08df63f\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " Oct 29 22:20:21.578836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.578647 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-tmp-dir\") pod \"cdaaa603-46cc-43ed-a102-9005c08df63f\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " Oct 29 22:20:21.578836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.578765 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-home\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-home\") pod \"cdaaa603-46cc-43ed-a102-9005c08df63f\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " Oct 29 22:20:21.579157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.578952 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdaaa603-46cc-43ed-a102-9005c08df63f-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdaaa603-46cc-43ed-a102-9005c08df63f-tls-certs\") pod \"cdaaa603-46cc-43ed-a102-9005c08df63f\" (UID: \"cdaaa603-46cc-43ed-a102-9005c08df63f\") " Oct 29 22:20:21.579157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.579095 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-model-cache" pod "cdaaa603-46cc-43ed-a102-9005c08df63f" (UID: "cdaaa603-46cc-43ed-a102-9005c08df63f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:21.579757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.579733 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:21.580009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.579987 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-tmp-dir" pod "cdaaa603-46cc-43ed-a102-9005c08df63f" (UID: "cdaaa603-46cc-43ed-a102-9005c08df63f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:21.581825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.581796 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdaaa603-46cc-43ed-a102-9005c08df63f-tls-certs" pod "cdaaa603-46cc-43ed-a102-9005c08df63f" (UID: "cdaaa603-46cc-43ed-a102-9005c08df63f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:20:21.582088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.582066 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdaaa603-46cc-43ed-a102-9005c08df63f-kube-api-access-vqshk" pod "cdaaa603-46cc-43ed-a102-9005c08df63f" (UID: "cdaaa603-46cc-43ed-a102-9005c08df63f"). InnerVolumeSpecName "kube-api-access-vqshk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:21.582440 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.582413 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-home" pod "cdaaa603-46cc-43ed-a102-9005c08df63f" (UID: "cdaaa603-46cc-43ed-a102-9005c08df63f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:21.582738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.582707 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-dshm" pod "cdaaa603-46cc-43ed-a102-9005c08df63f" (UID: "cdaaa603-46cc-43ed-a102-9005c08df63f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:21.651257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.647427 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:21.685921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.683966 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:21.685921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.683995 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:21.685921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.684004 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdaaa603-46cc-43ed-a102-9005c08df63f-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:21.685921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.684012 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqshk\" (UniqueName: \"kubernetes.io/projected/cdaaa603-46cc-43ed-a102-9005c08df63f-kube-api-access-vqshk\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:21.685921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:21.684022 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdaaa603-46cc-43ed-a102-9005c08df63f-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:22.279424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.279386 4952 generic.go:356] "Generic (PLEG): container finished" podID="cdaaa603-46cc-43ed-a102-9005c08df63f" containerID="68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292" exitCode=0 Oct 29 22:20:22.279424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.279423 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" Oct 29 22:20:22.279670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.279467 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" event={"ID":"cdaaa603-46cc-43ed-a102-9005c08df63f","Type":"ContainerDied","Data":"68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292"} Oct 29 22:20:22.279670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.279509 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n" event={"ID":"cdaaa603-46cc-43ed-a102-9005c08df63f","Type":"ContainerDied","Data":"08d1ec9275106a77aa2020241b0d2e7eceb6bae08ac09f1bb9e41a579ee995f6"} Oct 29 22:20:22.279670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.279530 4952 scope.go:122] "RemoveContainer" containerID="68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292" Oct 29 22:20:22.290791 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.290775 4952 scope.go:122] "RemoveContainer" containerID="68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292" Oct 29 22:20:22.291068 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:20:22.291047 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292\": container with ID starting with 68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292 not found: ID does not exist" containerID="68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292" Oct 29 22:20:22.291112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.291076 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292"} err="failed to get container status \"68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292\": rpc error: code = NotFound desc = could not find container \"68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292\": container with ID starting with 68ce378aaa3b87f94d31ecfadced04c66735516d1df829666c9dbd4b58088292 not found: ID does not exist" Oct 29 22:20:22.368227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.368191 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n"] Oct 29 22:20:22.378552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.378524 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-99e420/auth-test-model-99e420-kserve-f54c767c7-hmn6n"] Oct 29 22:20:22.480964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.480919 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:22.648499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.648457 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdaaa603-46cc-43ed-a102-9005c08df63f" path="/var/lib/kubelet/pods/cdaaa603-46cc-43ed-a102-9005c08df63f/volumes" Oct 29 22:20:22.660178 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.660144 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:22.830318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:22.830280 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:23.160310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:23.159421 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:23.338205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:23.338167 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:23.511445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:23.511362 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:23.985897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:23.985844 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:24.180929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:24.179552 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:25.495246 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:25.495210 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:25.680927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:25.677416 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:27.290727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:27.290657 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:28.853860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:28.853818 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:30.431746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:30.424833 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:31.583687 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:31.583637 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:20:31.898979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:31.893475 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6"] Oct 29 22:20:32.023822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:32.023780 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:32.461926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:32.455137 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:33.648928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:33.644671 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:34.154933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.147006 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:20:34.309313 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.309188 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:34.328025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.326986 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" podUID="edf26737-8758-4654-8213-3e0d7db85ee7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.136:8000/ready\": context deadline exceeded" Oct 29 22:20:34.349761 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.348646 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 22:20:34.466158 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.466056 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:20:34.497700 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.497663 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:34.499358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.499328 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:20:34.662045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.661999 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:20:34.707480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.707431 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:20:34.724985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.709634 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:20:34.734142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.734065 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:20:34.836938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.820581 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:20:34.836938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.828574 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 29 22:20:34.867928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:34.845566 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" podUID="d2c25e40-1aee-4a0c-aba8-ec071797192e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.139:8000/ready\": context deadline exceeded" Oct 29 22:20:35.050111 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:35.049997 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:20:35.693954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:35.693900 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:37.363012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:37.362965 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:38.814128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:38.814086 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:38.993764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:38.993727 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:39.173972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:39.173885 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:40.527106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:40.527071 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:40.716780 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:40.716743 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:41.136688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:41.136646 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:43.031120 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:43.031076 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:44.711934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:44.711535 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:45.540251 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:20:45.539419 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:20:46.494100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:46.494061 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:46.933439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:46.933370 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" podUID="d2c25e40-1aee-4a0c-aba8-ec071797192e" containerName="main" containerID="cri-o://09d2dacb89b92d40f3d053a9d1a678fe38cdc7c5148c5bf9ea03598c32da2d0b" gracePeriod=45 Oct 29 22:20:47.150579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.150539 4952 generic.go:356] "Generic (PLEG): container finished" podID="d2c25e40-1aee-4a0c-aba8-ec071797192e" containerID="09d2dacb89b92d40f3d053a9d1a678fe38cdc7c5148c5bf9ea03598c32da2d0b" exitCode=0 Oct 29 22:20:47.150848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.150702 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" event={"ID":"d2c25e40-1aee-4a0c-aba8-ec071797192e","Type":"ContainerDied","Data":"09d2dacb89b92d40f3d053a9d1a678fe38cdc7c5148c5bf9ea03598c32da2d0b"} Oct 29 22:20:47.386818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.386787 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:20:47.506448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.506366 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2c25e40-1aee-4a0c-aba8-ec071797192e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c25e40-1aee-4a0c-aba8-ec071797192e-tls-certs\") pod \"d2c25e40-1aee-4a0c-aba8-ec071797192e\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " Oct 29 22:20:47.506948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.506584 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-model-cache\") pod \"d2c25e40-1aee-4a0c-aba8-ec071797192e\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " Oct 29 22:20:47.506948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.506665 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c25e40-1aee-4a0c-aba8-ec071797192e-kube-api-access-2thbw\" (UniqueName: \"kubernetes.io/projected/d2c25e40-1aee-4a0c-aba8-ec071797192e-kube-api-access-2thbw\") pod \"d2c25e40-1aee-4a0c-aba8-ec071797192e\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " Oct 29 22:20:47.506948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.506712 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-home\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-home\") pod \"d2c25e40-1aee-4a0c-aba8-ec071797192e\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " Oct 29 22:20:47.506948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.506737 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-tmp-dir\") pod \"d2c25e40-1aee-4a0c-aba8-ec071797192e\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " Oct 29 22:20:47.506948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.506799 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-dshm\") pod \"d2c25e40-1aee-4a0c-aba8-ec071797192e\" (UID: \"d2c25e40-1aee-4a0c-aba8-ec071797192e\") " Oct 29 22:20:47.507277 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.507056 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-home" pod "d2c25e40-1aee-4a0c-aba8-ec071797192e" (UID: "d2c25e40-1aee-4a0c-aba8-ec071797192e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:47.507277 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.507083 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-tmp-dir" pod "d2c25e40-1aee-4a0c-aba8-ec071797192e" (UID: "d2c25e40-1aee-4a0c-aba8-ec071797192e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:47.507484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.507463 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:47.507576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.507484 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:47.507576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.507541 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-model-cache" pod "d2c25e40-1aee-4a0c-aba8-ec071797192e" (UID: "d2c25e40-1aee-4a0c-aba8-ec071797192e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:47.510274 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.510249 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-dshm" pod "d2c25e40-1aee-4a0c-aba8-ec071797192e" (UID: "d2c25e40-1aee-4a0c-aba8-ec071797192e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:20:47.510428 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.510356 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2c25e40-1aee-4a0c-aba8-ec071797192e-tls-certs" pod "d2c25e40-1aee-4a0c-aba8-ec071797192e" (UID: "d2c25e40-1aee-4a0c-aba8-ec071797192e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:20:47.510504 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.510479 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c25e40-1aee-4a0c-aba8-ec071797192e-kube-api-access-2thbw" pod "d2c25e40-1aee-4a0c-aba8-ec071797192e" (UID: "d2c25e40-1aee-4a0c-aba8-ec071797192e"). InnerVolumeSpecName "kube-api-access-2thbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:20:47.609368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.609336 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:47.609368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.609368 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2thbw\" (UniqueName: \"kubernetes.io/projected/d2c25e40-1aee-4a0c-aba8-ec071797192e-kube-api-access-2thbw\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:47.609573 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.609382 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c25e40-1aee-4a0c-aba8-ec071797192e-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:47.609573 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:47.609393 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c25e40-1aee-4a0c-aba8-ec071797192e-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:20:48.162429 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:48.162390 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" event={"ID":"d2c25e40-1aee-4a0c-aba8-ec071797192e","Type":"ContainerDied","Data":"7589eafde4a4d6c429211bf1ed645425c0844e02ff314d7121c2a2818fc90615"} Oct 29 22:20:48.162628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:48.162457 4952 scope.go:122] "RemoveContainer" containerID="09d2dacb89b92d40f3d053a9d1a678fe38cdc7c5148c5bf9ea03598c32da2d0b" Oct 29 22:20:48.162628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:48.162619 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6" Oct 29 22:20:48.250067 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:48.247478 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6"] Oct 29 22:20:48.297002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:48.296957 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-15154b/auth-test-model-15154b-kserve-7f6bcdb45-gfpj6"] Oct 29 22:20:48.300879 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:48.300803 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:48.311382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:48.311349 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct"] Oct 29 22:20:48.487672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:48.487572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:48.651921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:48.649349 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c25e40-1aee-4a0c-aba8-ec071797192e" path="/var/lib/kubelet/pods/d2c25e40-1aee-4a0c-aba8-ec071797192e/volumes" Oct 29 22:20:49.978094 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:49.978058 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:50.177614 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:50.175632 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:50.531928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:50.530327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:50.837928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:50.837643 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:52.081470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:52.081426 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:52.634926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:52.630097 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:53.838751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:53.838709 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:54.034929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:54.034431 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:54.216863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:54.216769 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:54.405932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:54.402461 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:54.587682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:54.587637 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:54.773750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:54.773701 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:55.378053 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:55.378014 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:55.576929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:55.574793 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:56.158393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:56.158356 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:56.653941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:56.617969 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:57.239577 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:57.239540 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:57.421962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:57.421831 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:59.447869 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:20:59.447829 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:01.121117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:01.121074 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:02.732926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:02.732596 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:03.353971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.353924 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" podUID="edf26737-8758-4654-8213-3e0d7db85ee7" containerName="main" containerID="cri-o://a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d" gracePeriod=45 Oct 29 22:21:03.674392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.674362 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:21:03.774153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.774115 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf26737-8758-4654-8213-3e0d7db85ee7-kube-api-access-qnp4s\" (UniqueName: \"kubernetes.io/projected/edf26737-8758-4654-8213-3e0d7db85ee7-kube-api-access-qnp4s\") pod \"edf26737-8758-4654-8213-3e0d7db85ee7\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " Oct 29 22:21:03.774563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.774217 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-dshm\") pod \"edf26737-8758-4654-8213-3e0d7db85ee7\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " Oct 29 22:21:03.774563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.774378 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-model-cache\") pod \"edf26737-8758-4654-8213-3e0d7db85ee7\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " Oct 29 22:21:03.774563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.774502 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-tmp-dir\") pod \"edf26737-8758-4654-8213-3e0d7db85ee7\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " Oct 29 22:21:03.775289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.774591 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-home\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-home\") pod \"edf26737-8758-4654-8213-3e0d7db85ee7\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " Oct 29 22:21:03.775289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.774636 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf26737-8758-4654-8213-3e0d7db85ee7-tls-certs\" (UniqueName: \"kubernetes.io/secret/edf26737-8758-4654-8213-3e0d7db85ee7-tls-certs\") pod \"edf26737-8758-4654-8213-3e0d7db85ee7\" (UID: \"edf26737-8758-4654-8213-3e0d7db85ee7\") " Oct 29 22:21:03.775289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.774666 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-model-cache" pod "edf26737-8758-4654-8213-3e0d7db85ee7" (UID: "edf26737-8758-4654-8213-3e0d7db85ee7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:03.775289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.774765 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-tmp-dir" pod "edf26737-8758-4654-8213-3e0d7db85ee7" (UID: "edf26737-8758-4654-8213-3e0d7db85ee7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:03.775289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.774866 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-home" pod "edf26737-8758-4654-8213-3e0d7db85ee7" (UID: "edf26737-8758-4654-8213-3e0d7db85ee7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:03.775508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.775426 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:03.775508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.775453 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:03.775508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.775467 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:03.776783 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.776725 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-dshm" pod "edf26737-8758-4654-8213-3e0d7db85ee7" (UID: "edf26737-8758-4654-8213-3e0d7db85ee7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:03.776936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.776917 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf26737-8758-4654-8213-3e0d7db85ee7-tls-certs" pod "edf26737-8758-4654-8213-3e0d7db85ee7" (UID: "edf26737-8758-4654-8213-3e0d7db85ee7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:21:03.777047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.777031 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf26737-8758-4654-8213-3e0d7db85ee7-kube-api-access-qnp4s" pod "edf26737-8758-4654-8213-3e0d7db85ee7" (UID: "edf26737-8758-4654-8213-3e0d7db85ee7"). InnerVolumeSpecName "kube-api-access-qnp4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:21:03.876690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.876656 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf26737-8758-4654-8213-3e0d7db85ee7-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:03.876690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.876687 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnp4s\" (UniqueName: \"kubernetes.io/projected/edf26737-8758-4654-8213-3e0d7db85ee7-kube-api-access-qnp4s\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:03.876690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:03.876697 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf26737-8758-4654-8213-3e0d7db85ee7-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:04.368774 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.368735 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:04.419573 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.419532 4952 generic.go:356] "Generic (PLEG): container finished" podID="edf26737-8758-4654-8213-3e0d7db85ee7" containerID="a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d" exitCode=0 Oct 29 22:21:04.419812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.419792 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" event={"ID":"edf26737-8758-4654-8213-3e0d7db85ee7","Type":"ContainerDied","Data":"a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d"} Oct 29 22:21:04.419887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.419833 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" event={"ID":"edf26737-8758-4654-8213-3e0d7db85ee7","Type":"ContainerDied","Data":"777892fc7143a891d8350e0fe5d92f0e83557e8d5e97b35e0934238124450b17"} Oct 29 22:21:04.419887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.419855 4952 scope.go:122] "RemoveContainer" containerID="a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d" Oct 29 22:21:04.420085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.420070 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct" Oct 29 22:21:04.437266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.437239 4952 scope.go:122] "RemoveContainer" containerID="a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d" Oct 29 22:21:04.437659 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:21:04.437627 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d\": container with ID starting with a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d not found: ID does not exist" containerID="a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d" Oct 29 22:21:04.437769 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.437667 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d"} err="failed to get container status \"a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d\": rpc error: code = NotFound desc = could not find container \"a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d\": container with ID starting with a60eb810ab63bdbecfdfd98d299f31a03854140d8a3df7ae7d546dbe3463c88d not found: ID does not exist" Oct 29 22:21:04.480557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.480519 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct"] Oct 29 22:21:04.492159 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.492124 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-76c1b7/auth-test-model-76c1b7-kserve-668977b7b5-6x9ct"] Oct 29 22:21:04.647175 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:04.647090 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf26737-8758-4654-8213-3e0d7db85ee7" path="/var/lib/kubelet/pods/edf26737-8758-4654-8213-3e0d7db85ee7/volumes" Oct 29 22:21:06.003564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:06.003471 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:07.625928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:07.625176 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:09.255929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:09.255858 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:10.227466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:10.227429 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:10.829206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:10.829105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:12.542025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:12.538363 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:14.142661 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:14.142619 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:15.571682 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:21:15.571635 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:21:15.731556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:15.726966 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:18.319369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:18.319337 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:18.939221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:18.939180 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:20.631339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:20.631298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:21.670244 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:21.670207 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:21.843732 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:21.843693 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:22.209636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:22.209599 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:23.761555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:23.761516 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:25.277523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.277490 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5"] Oct 29 22:21:25.278199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278183 4952 cpu_manager.go:414] "containerMap: removing container" podUID="edf26737-8758-4654-8213-3e0d7db85ee7" containerName="main" Oct 29 22:21:25.278261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278200 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf26737-8758-4654-8213-3e0d7db85ee7" containerName="main" Oct 29 22:21:25.278261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278235 4952 cpu_manager.go:414] "containerMap: removing container" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" Oct 29 22:21:25.278261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278240 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" Oct 29 22:21:25.278261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278253 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" Oct 29 22:21:25.278261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278258 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" Oct 29 22:21:25.278426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278275 4952 cpu_manager.go:414] "containerMap: removing container" podUID="cdaaa603-46cc-43ed-a102-9005c08df63f" containerName="main" Oct 29 22:21:25.278426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278281 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdaaa603-46cc-43ed-a102-9005c08df63f" containerName="main" Oct 29 22:21:25.278426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278291 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d2c25e40-1aee-4a0c-aba8-ec071797192e" containerName="main" Oct 29 22:21:25.278426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278295 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c25e40-1aee-4a0c-aba8-ec071797192e" containerName="main" Oct 29 22:21:25.278426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278399 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf26737-8758-4654-8213-3e0d7db85ee7" containerName="main" Oct 29 22:21:25.278426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278413 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="55eb0002-813c-4f3f-9951-4bbe18a8802c" containerName="istio-proxy" Oct 29 22:21:25.278426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278419 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dcaf3ea-6b6e-4d3d-865d-55987e5f9446" containerName="istio-proxy" Oct 29 22:21:25.278426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278426 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdaaa603-46cc-43ed-a102-9005c08df63f" containerName="main" Oct 29 22:21:25.278663 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.278433 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c25e40-1aee-4a0c-aba8-ec071797192e" containerName="main" Oct 29 22:21:25.283846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.283819 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.300112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.300081 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-7fe78e-serving-cert\"" Oct 29 22:21:25.303845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.303821 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5"] Oct 29 22:21:25.396899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.396860 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f6c96a8-b18d-4f10-8147-34df971af0c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.397101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.397021 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffzmt\" (UniqueName: \"kubernetes.io/projected/4f6c96a8-b18d-4f10-8147-34df971af0c3-kube-api-access-ffzmt\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.397144 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.397131 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.397208 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.397197 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.405157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.405130 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:25.505925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.502022 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f6c96a8-b18d-4f10-8147-34df971af0c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.505925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.502147 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffzmt\" (UniqueName: \"kubernetes.io/projected/4f6c96a8-b18d-4f10-8147-34df971af0c3-kube-api-access-ffzmt\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.505925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.502261 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.505925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.502350 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.507620 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.507583 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f6c96a8-b18d-4f10-8147-34df971af0c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.515248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.515213 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.560510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.555166 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffzmt\" (UniqueName: \"kubernetes.io/projected/4f6c96a8-b18d-4f10-8147-34df971af0c3-kube-api-access-ffzmt\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.560510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.556614 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.575167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.575138 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:25.601688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.601659 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:25.933601 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:21:25.933570 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6c96a8_b18d_4f10_8147_34df971af0c3.slice/crio-b43b300daae8fd7ca62bcfb70045d8f2ebed459a8ca63c7caccaa5f673906844 WatchSource:0}: Error finding container b43b300daae8fd7ca62bcfb70045d8f2ebed459a8ca63c7caccaa5f673906844: Status 404 returned error can't find the container with id b43b300daae8fd7ca62bcfb70045d8f2ebed459a8ca63c7caccaa5f673906844 Oct 29 22:21:25.934397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:25.934367 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5"] Oct 29 22:21:26.634210 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:26.634171 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" event={"ID":"4f6c96a8-b18d-4f10-8147-34df971af0c3","Type":"ContainerStarted","Data":"67914fcb09125fdd25fd723adbf5ef9f166e604bb58a441ccb6cd0b4ba9cd879"} Oct 29 22:21:26.634696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:26.634242 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:26.634696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:26.634269 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" event={"ID":"4f6c96a8-b18d-4f10-8147-34df971af0c3","Type":"ContainerStarted","Data":"b43b300daae8fd7ca62bcfb70045d8f2ebed459a8ca63c7caccaa5f673906844"} Oct 29 22:21:26.678609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:26.678549 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" podStartSLOduration=1.303873269 podStartE2EDuration="1.678534435s" totalImagesPullingTime="374.661166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:26.676360182 +0000 UTC m=+2352.150948600" watchObservedRunningTime="2026-10-29 22:21:26.678534435 +0000 UTC m=+2352.153123240" Oct 29 22:21:27.190282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:27.189949 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:28.824928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:28.824510 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:29.326393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:29.326361 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x"] Oct 29 22:21:29.326617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:29.326590 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" containerID="cri-o://e4ef3ad7d6d5e2e036921bba20b359c1735f8dac89406bcff42fd8f9cd5676c3" gracePeriod=30 Oct 29 22:21:30.378930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:30.378185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:32.037540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:32.037505 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:32.798965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:32.797425 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:21:34.546005 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:34.531849 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:34.904044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:34.903670 4952 generic.go:356] "Generic (PLEG): container finished" podID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerID="e4ef3ad7d6d5e2e036921bba20b359c1735f8dac89406bcff42fd8f9cd5676c3" exitCode=0 Oct 29 22:21:34.904044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:34.903969 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" event={"ID":"99e98d47-7a58-41d8-8ab2-09b52addd0ce","Type":"ContainerDied","Data":"e4ef3ad7d6d5e2e036921bba20b359c1735f8dac89406bcff42fd8f9cd5676c3"} Oct 29 22:21:35.034744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.034711 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:21:35.126923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.126867 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-certs\") pod \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " Oct 29 22:21:35.127108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.126995 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-socket\") pod \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " Oct 29 22:21:35.127167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.127113 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-kube-api-access-zxh5d\" (UniqueName: \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-kube-api-access-zxh5d\") pod \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " Oct 29 22:21:35.127264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.127245 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-podinfo\") pod \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " Oct 29 22:21:35.127318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.127277 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-envoy\") pod \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " Oct 29 22:21:35.127318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.127314 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-token\" (UniqueName: \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-token\") pod \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " Oct 29 22:21:35.127407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.127344 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-credential-socket\") pod \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " Oct 29 22:21:35.127407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.127392 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-data\") pod \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " Oct 29 22:21:35.127511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.127497 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istiod-ca-cert\") pod \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\" (UID: \"99e98d47-7a58-41d8-8ab2-09b52addd0ce\") " Oct 29 22:21:35.128556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.128521 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istiod-ca-cert" pod "99e98d47-7a58-41d8-8ab2-09b52addd0ce" (UID: "99e98d47-7a58-41d8-8ab2-09b52addd0ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:21:35.128821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.128800 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-certs" pod "99e98d47-7a58-41d8-8ab2-09b52addd0ce" (UID: "99e98d47-7a58-41d8-8ab2-09b52addd0ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:35.129031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.129010 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-socket" pod "99e98d47-7a58-41d8-8ab2-09b52addd0ce" (UID: "99e98d47-7a58-41d8-8ab2-09b52addd0ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:35.129220 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.129199 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-credential-socket" pod "99e98d47-7a58-41d8-8ab2-09b52addd0ce" (UID: "99e98d47-7a58-41d8-8ab2-09b52addd0ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:35.129220 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.129210 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-data" pod "99e98d47-7a58-41d8-8ab2-09b52addd0ce" (UID: "99e98d47-7a58-41d8-8ab2-09b52addd0ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:35.133816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.133785 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-token" pod "99e98d47-7a58-41d8-8ab2-09b52addd0ce" (UID: "99e98d47-7a58-41d8-8ab2-09b52addd0ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:21:35.135225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.135057 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-podinfo" pod "99e98d47-7a58-41d8-8ab2-09b52addd0ce" (UID: "99e98d47-7a58-41d8-8ab2-09b52addd0ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:21:35.136034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.135984 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-kube-api-access-zxh5d" pod "99e98d47-7a58-41d8-8ab2-09b52addd0ce" (UID: "99e98d47-7a58-41d8-8ab2-09b52addd0ce"). InnerVolumeSpecName "kube-api-access-zxh5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:21:35.136460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.136432 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-envoy" pod "99e98d47-7a58-41d8-8ab2-09b52addd0ce" (UID: "99e98d47-7a58-41d8-8ab2-09b52addd0ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:35.229681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.229648 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxh5d\" (UniqueName: \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-kube-api-access-zxh5d\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:35.229681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.229680 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:35.229681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.229689 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:35.229958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.229697 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:35.229958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.229706 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:35.229958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.229717 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:35.229958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.229730 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e98d47-7a58-41d8-8ab2-09b52addd0ce-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:35.229958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.229740 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:35.229958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.229749 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e98d47-7a58-41d8-8ab2-09b52addd0ce-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:35.917505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.917465 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" event={"ID":"99e98d47-7a58-41d8-8ab2-09b52addd0ce","Type":"ContainerDied","Data":"8b2bf5433102494817cf18958fd0f36680f910d27222721bddd76bbf4375bc8f"} Oct 29 22:21:35.918011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.917540 4952 scope.go:122] "RemoveContainer" containerID="e4ef3ad7d6d5e2e036921bba20b359c1735f8dac89406bcff42fd8f9cd5676c3" Oct 29 22:21:35.918011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.917715 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x" Oct 29 22:21:35.996843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:35.996810 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x"] Oct 29 22:21:36.008164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:36.008131 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-fjd5x"] Oct 29 22:21:36.235819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:36.235721 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:36.647698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:36.647660 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" path="/var/lib/kubelet/pods/99e98d47-7a58-41d8-8ab2-09b52addd0ce/volumes" Oct 29 22:21:37.669482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.669450 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7770-5f714d"] Oct 29 22:21:37.670375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.670354 4952 cpu_manager.go:414] "containerMap: removing container" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" Oct 29 22:21:37.670437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.670376 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" Oct 29 22:21:37.670536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.670517 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e98d47-7a58-41d8-8ab2-09b52addd0ce" containerName="istio-proxy" Oct 29 22:21:37.675530 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.675508 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-5f714d" Oct 29 22:21:37.693117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.693092 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7770-5f714d"] Oct 29 22:21:37.698971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.698940 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r84ws\"" Oct 29 22:21:37.784861 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.784828 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tjz\" (UniqueName: \"kubernetes.io/projected/360e590b-6892-4a84-abd8-3902cc57f96b-kube-api-access-96tjz\") pod \"test-curl-7770-5f714d\" (UID: \"360e590b-6892-4a84-abd8-3902cc57f96b\") " pod="opendatahub/test-curl-7770-5f714d" Oct 29 22:21:37.845928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.845384 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:37.886816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.886781 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tjz\" (UniqueName: \"kubernetes.io/projected/360e590b-6892-4a84-abd8-3902cc57f96b-kube-api-access-96tjz\") pod \"test-curl-7770-5f714d\" (UID: \"360e590b-6892-4a84-abd8-3902cc57f96b\") " pod="opendatahub/test-curl-7770-5f714d" Oct 29 22:21:37.912104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.912067 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tjz\" (UniqueName: \"kubernetes.io/projected/360e590b-6892-4a84-abd8-3902cc57f96b-kube-api-access-96tjz\") pod \"test-curl-7770-5f714d\" (UID: \"360e590b-6892-4a84-abd8-3902cc57f96b\") " pod="opendatahub/test-curl-7770-5f714d" Oct 29 22:21:37.989949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:37.989857 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-5f714d" Oct 29 22:21:38.207454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:38.207416 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:38.306228 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:38.306193 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7770-5f714d"] Oct 29 22:21:38.306691 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:21:38.306667 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod360e590b_6892_4a84_abd8_3902cc57f96b.slice/crio-cd85b705feeea7f2c15276a9b03cad4ea8e1630417f61325bace1eb6bf1cdfa9 WatchSource:0}: Error finding container cd85b705feeea7f2c15276a9b03cad4ea8e1630417f61325bace1eb6bf1cdfa9: Status 404 returned error can't find the container with id cd85b705feeea7f2c15276a9b03cad4ea8e1630417f61325bace1eb6bf1cdfa9 Oct 29 22:21:38.943985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:38.943945 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7770-5f714d" event={"ID":"360e590b-6892-4a84-abd8-3902cc57f96b","Type":"ContainerStarted","Data":"b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a"} Oct 29 22:21:38.944401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:38.944002 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7770-5f714d" event={"ID":"360e590b-6892-4a84-abd8-3902cc57f96b","Type":"ContainerStarted","Data":"cd85b705feeea7f2c15276a9b03cad4ea8e1630417f61325bace1eb6bf1cdfa9"} Oct 29 22:21:38.985001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:38.984953 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7770-5f714d" podStartSLOduration=1.984939238 podStartE2EDuration="1.984939238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:38.982234222 +0000 UTC m=+2364.456822639" watchObservedRunningTime="2026-10-29 22:21:38.984939238 +0000 UTC m=+2364.459527661" Oct 29 22:21:39.421927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:39.420533 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:39.908895 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:39.908853 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7770-5f714d"] Oct 29 22:21:39.920609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:39.920567 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7770-5f714d"] Oct 29 22:21:40.953414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:40.953374 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm"] Oct 29 22:21:40.954434 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:40.954416 4952 cpu_manager.go:414] "containerMap: removing container" podUID="360e590b-6892-4a84-abd8-3902cc57f96b" containerName="test-curl-7770-5f714d" Oct 29 22:21:40.954492 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:40.954438 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360e590b-6892-4a84-abd8-3902cc57f96b" containerName="test-curl-7770-5f714d" Oct 29 22:21:40.954635 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:40.954624 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="360e590b-6892-4a84-abd8-3902cc57f96b" containerName="test-curl-7770-5f714d" Oct 29 22:21:40.958889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:40.958861 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:40.963204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:40.963170 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7770-5f714d" podUID="360e590b-6892-4a84-abd8-3902cc57f96b" containerName="test-curl-7770-5f714d" containerID="cri-o://b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a" gracePeriod=2 Oct 29 22:21:40.979876 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:40.979840 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm"] Oct 29 22:21:40.980472 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:40.980166 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-b9ed94-serving-cert\"" Oct 29 22:21:41.053807 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.053769 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.054071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.053961 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-metrics-certs\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.054197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.054169 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfspl\" (UniqueName: \"kubernetes.io/projected/a8883ff6-916d-43bd-a221-c8cea4992968-kube-api-access-rfspl\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.054450 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.054417 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8883ff6-916d-43bd-a221-c8cea4992968-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.091062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.091006 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:41.155770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.155724 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8883ff6-916d-43bd-a221-c8cea4992968-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.155953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.155824 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.155953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.155919 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-metrics-certs\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.156078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.155980 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfspl\" (UniqueName: \"kubernetes.io/projected/a8883ff6-916d-43bd-a221-c8cea4992968-kube-api-access-rfspl\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.156433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.156399 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8883ff6-916d-43bd-a221-c8cea4992968-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.158724 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.158701 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.158926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.158888 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-metrics-certs\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.212793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.195794 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfspl\" (UniqueName: \"kubernetes.io/projected/a8883ff6-916d-43bd-a221-c8cea4992968-kube-api-access-rfspl\") pod \"maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.264483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.261476 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:41.277006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.276972 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:41.594888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.594860 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm"] Oct 29 22:21:41.596034 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:21:41.596008 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8883ff6_916d_43bd_a221_c8cea4992968.slice/crio-2635f655f4b2198a167c87ca80c7127f3df59e60023ebfa8454041f9e4ae4bd5 WatchSource:0}: Error finding container 2635f655f4b2198a167c87ca80c7127f3df59e60023ebfa8454041f9e4ae4bd5: Status 404 returned error can't find the container with id 2635f655f4b2198a167c87ca80c7127f3df59e60023ebfa8454041f9e4ae4bd5 Oct 29 22:21:41.975081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:41.975041 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" event={"ID":"a8883ff6-916d-43bd-a221-c8cea4992968","Type":"ContainerStarted","Data":"2635f655f4b2198a167c87ca80c7127f3df59e60023ebfa8454041f9e4ae4bd5"} Oct 29 22:21:42.925006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:42.923635 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:42.990089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:42.990049 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" event={"ID":"a8883ff6-916d-43bd-a221-c8cea4992968","Type":"ContainerStarted","Data":"c9c14914614a840b89c1a46b2e511f3ece3d2d88bbbea4f36d76e41a41047eed"} Oct 29 22:21:42.990456 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:42.990180 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:43.036978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:43.036895 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" podStartSLOduration=2.672541234 podStartE2EDuration="3.036880254s" totalImagesPullingTime="364.33902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:43.03317266 +0000 UTC m=+2368.507761079" watchObservedRunningTime="2026-10-29 22:21:43.036880254 +0000 UTC m=+2368.511468672" Oct 29 22:21:43.277389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:43.277360 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-5f714d" Oct 29 22:21:43.310408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:43.310377 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360e590b-6892-4a84-abd8-3902cc57f96b-kube-api-access-96tjz\" (UniqueName: \"kubernetes.io/projected/360e590b-6892-4a84-abd8-3902cc57f96b-kube-api-access-96tjz\") pod \"360e590b-6892-4a84-abd8-3902cc57f96b\" (UID: \"360e590b-6892-4a84-abd8-3902cc57f96b\") " Oct 29 22:21:43.312623 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:43.312590 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360e590b-6892-4a84-abd8-3902cc57f96b-kube-api-access-96tjz" pod "360e590b-6892-4a84-abd8-3902cc57f96b" (UID: "360e590b-6892-4a84-abd8-3902cc57f96b"). InnerVolumeSpecName "kube-api-access-96tjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:21:43.412370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:43.412335 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96tjz\" (UniqueName: \"kubernetes.io/projected/360e590b-6892-4a84-abd8-3902cc57f96b-kube-api-access-96tjz\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:21:43.999260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:43.999219 4952 generic.go:356] "Generic (PLEG): container finished" podID="360e590b-6892-4a84-abd8-3902cc57f96b" containerID="b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a" exitCode=137 Oct 29 22:21:43.999737 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:43.999263 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-5f714d" Oct 29 22:21:43.999737 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:43.999314 4952 scope.go:122] "RemoveContainer" containerID="b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a" Oct 29 22:21:44.010436 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:44.010414 4952 scope.go:122] "RemoveContainer" containerID="b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a" Oct 29 22:21:44.010732 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:21:44.010709 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a\": container with ID starting with b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a not found: ID does not exist" containerID="b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a" Oct 29 22:21:44.010803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:44.010745 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a"} err="failed to get container status \"b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a\": rpc error: code = NotFound desc = could not find container \"b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a\": container with ID starting with b347b494b5e376165c79f8731dd05ab7ef7356ef834bc851f2e539fade2d705a not found: ID does not exist" Oct 29 22:21:44.021684 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:21:44.021660 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7770-5f714d\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="360e590b-6892-4a84-abd8-3902cc57f96b" pod="opendatahub/test-curl-7770-5f714d" Oct 29 22:21:44.553948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:44.553623 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:44.648318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:44.648281 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360e590b-6892-4a84-abd8-3902cc57f96b" path="/var/lib/kubelet/pods/360e590b-6892-4a84-abd8-3902cc57f96b/volumes" Oct 29 22:21:44.655156 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:21:44.655118 4952 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7770-5f714d\" is forbidden: User \"system:node:ip-10-0-130-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-134.ec2.internal' and this object" podUID="360e590b-6892-4a84-abd8-3902cc57f96b" pod="opendatahub/test-curl-7770-5f714d" Oct 29 22:21:45.530209 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:21:45.530173 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:21:46.119004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:46.117900 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:47.834297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:47.834038 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:49.005703 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:49.005673 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:21:49.455351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:49.455317 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:51.017445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:51.017408 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:52.641759 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:52.641708 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:54.180177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:54.180140 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:55.724789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:55.724743 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:57.413113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:57.413072 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:59.046927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:21:59.046728 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:01.451628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:01.451575 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:02.173445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:02.173391 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:02.721100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:02.721055 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:03.873152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:03.873081 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:04.960393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:04.960355 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:22:05.541930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:05.538016 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:07.634926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:07.621024 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:08.748685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:08.748639 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r"] Oct 29 22:22:08.749266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:08.749069 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" containerID="cri-o://8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e" gracePeriod=30 Oct 29 22:22:09.334463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:09.334419 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:10.360536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:10.360501 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:11.007514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:11.007472 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:12.659475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:12.651532 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:14.341175 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.341104 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:22:14.345097 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.345062 4952 generic.go:356] "Generic (PLEG): container finished" podID="73907711-dc9d-438f-b43f-a0385e6402be" containerID="8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e" exitCode=0 Oct 29 22:22:14.345266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.345148 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" Oct 29 22:22:14.345266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.345210 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" event={"ID":"73907711-dc9d-438f-b43f-a0385e6402be","Type":"ContainerDied","Data":"8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e"} Oct 29 22:22:14.345266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.345258 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r" event={"ID":"73907711-dc9d-438f-b43f-a0385e6402be","Type":"ContainerDied","Data":"779050673f5e238ac9fbfbe7bf2c1cc6679ade32db78b85dec431378fb8c9e95"} Oct 29 22:22:14.345390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.345288 4952 scope.go:122] "RemoveContainer" containerID="8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e" Oct 29 22:22:14.358651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.358633 4952 scope.go:122] "RemoveContainer" containerID="8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e" Oct 29 22:22:14.359526 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:22:14.358897 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e\": container with ID starting with 8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e not found: ID does not exist" containerID="8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e" Oct 29 22:22:14.359526 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.358979 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e"} err="failed to get container status \"8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e\": rpc error: code = NotFound desc = could not find container \"8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e\": container with ID starting with 8a9bdd9c9f4c8639f9c5d5b8cb8bc6b1384504703cffccf783cfc15ee6ef764e not found: ID does not exist" Oct 29 22:22:14.415553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.415517 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-socket\") pod \"73907711-dc9d-438f-b43f-a0385e6402be\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " Oct 29 22:22:14.415726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.415592 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73907711-dc9d-438f-b43f-a0385e6402be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73907711-dc9d-438f-b43f-a0385e6402be-istio-podinfo\") pod \"73907711-dc9d-438f-b43f-a0385e6402be\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " Oct 29 22:22:14.415726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.415628 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-credential-socket\") pod \"73907711-dc9d-438f-b43f-a0385e6402be\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " Oct 29 22:22:14.415811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.415774 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-kube-api-access-zgncb\" (UniqueName: \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-kube-api-access-zgncb\") pod \"73907711-dc9d-438f-b43f-a0385e6402be\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " Oct 29 22:22:14.415882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.415870 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-istio-token\" (UniqueName: \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-istio-token\") pod \"73907711-dc9d-438f-b43f-a0385e6402be\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " Oct 29 22:22:14.415957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.415894 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-data\") pod \"73907711-dc9d-438f-b43f-a0385e6402be\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " Oct 29 22:22:14.416003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.415970 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73907711-dc9d-438f-b43f-a0385e6402be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73907711-dc9d-438f-b43f-a0385e6402be-istiod-ca-cert\") pod \"73907711-dc9d-438f-b43f-a0385e6402be\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " Oct 29 22:22:14.416003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.415998 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-certs\") pod \"73907711-dc9d-438f-b43f-a0385e6402be\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " Oct 29 22:22:14.416080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.416023 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-envoy\") pod \"73907711-dc9d-438f-b43f-a0385e6402be\" (UID: \"73907711-dc9d-438f-b43f-a0385e6402be\") " Oct 29 22:22:14.417170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.417136 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-data" pod "73907711-dc9d-438f-b43f-a0385e6402be" (UID: "73907711-dc9d-438f-b43f-a0385e6402be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:14.417408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.417380 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-certs" pod "73907711-dc9d-438f-b43f-a0385e6402be" (UID: "73907711-dc9d-438f-b43f-a0385e6402be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:14.417511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.417445 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73907711-dc9d-438f-b43f-a0385e6402be-istiod-ca-cert" pod "73907711-dc9d-438f-b43f-a0385e6402be" (UID: "73907711-dc9d-438f-b43f-a0385e6402be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:22:14.417511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.417468 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-socket" pod "73907711-dc9d-438f-b43f-a0385e6402be" (UID: "73907711-dc9d-438f-b43f-a0385e6402be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:14.417627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.417590 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-credential-socket" pod "73907711-dc9d-438f-b43f-a0385e6402be" (UID: "73907711-dc9d-438f-b43f-a0385e6402be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:14.419623 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.419585 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73907711-dc9d-438f-b43f-a0385e6402be-istio-podinfo" pod "73907711-dc9d-438f-b43f-a0385e6402be" (UID: "73907711-dc9d-438f-b43f-a0385e6402be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:22:14.422194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.422168 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-istio-token" pod "73907711-dc9d-438f-b43f-a0385e6402be" (UID: "73907711-dc9d-438f-b43f-a0385e6402be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:14.424499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.424472 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-envoy" pod "73907711-dc9d-438f-b43f-a0385e6402be" (UID: "73907711-dc9d-438f-b43f-a0385e6402be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:14.424760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.424734 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-kube-api-access-zgncb" pod "73907711-dc9d-438f-b43f-a0385e6402be" (UID: "73907711-dc9d-438f-b43f-a0385e6402be"). InnerVolumeSpecName "kube-api-access-zgncb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:14.561931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.561822 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:14.563287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.563247 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgncb\" (UniqueName: \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-kube-api-access-zgncb\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.563287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.563279 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73907711-dc9d-438f-b43f-a0385e6402be-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.563501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.563297 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.563501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.563311 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73907711-dc9d-438f-b43f-a0385e6402be-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.563501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.563322 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.563501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.563334 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.563501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.563345 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.563501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.563358 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73907711-dc9d-438f-b43f-a0385e6402be-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.563501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.563370 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73907711-dc9d-438f-b43f-a0385e6402be-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:14.712786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.712740 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r"] Oct 29 22:22:14.728948 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:14.728899 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-97n2r"] Oct 29 22:22:15.461874 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:15.461838 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:22:15.470235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:15.470206 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:22:15.534972 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:22:15.534930 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:22:16.183304 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:16.174709 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:16.354654 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:16.354617 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:16.648539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:16.648502 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73907711-dc9d-438f-b43f-a0385e6402be" path="/var/lib/kubelet/pods/73907711-dc9d-438f-b43f-a0385e6402be/volumes" Oct 29 22:22:17.206970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:17.202301 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t"] Oct 29 22:22:17.308929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:17.303702 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:17.602618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:17.602550 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:18.037987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:18.032886 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:18.332977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:18.318410 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:18.533922 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:18.533864 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:18.939127 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:18.939091 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:19.454215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.454156 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm"] Oct 29 22:22:19.455342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.455310 4952 cpu_manager.go:414] "containerMap: removing container" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" Oct 29 22:22:19.455342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.455340 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" Oct 29 22:22:19.465923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.456758 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="73907711-dc9d-438f-b43f-a0385e6402be" containerName="istio-proxy" Oct 29 22:22:19.465923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.462874 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.490984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.488095 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-50d4af-openshift-default-dockercfg-6cvww\"" Oct 29 22:22:19.551283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.551256 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm"] Oct 29 22:22:19.555887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.555854 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:19.558439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.558363 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-credential-socket\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.558648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.558629 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-certs\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.558924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.558886 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3ce13d-da21-4593-8eba-603813119eea-istiod-ca-cert\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.559131 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.559114 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-istio-token\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.559354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.559336 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-data\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.559632 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.559611 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gkr\" (UniqueName: \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-kube-api-access-m2gkr\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.560041 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.560020 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-envoy\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.560284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.560261 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-socket\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.560458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.560440 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3ce13d-da21-4593-8eba-603813119eea-istio-podinfo\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.708099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.707953 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-socket\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.713250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.711866 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3ce13d-da21-4593-8eba-603813119eea-istio-podinfo\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.713622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.713594 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-credential-socket\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.744797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.743355 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-certs\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.744797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.743499 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3ce13d-da21-4593-8eba-603813119eea-istiod-ca-cert\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.744797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.743576 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-istio-token\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.744797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.743677 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-data\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.744797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.743882 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gkr\" (UniqueName: \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-kube-api-access-m2gkr\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.744797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.710352 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-socket\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.766379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.718347 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-credential-socket\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.766379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.758758 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3ce13d-da21-4593-8eba-603813119eea-istio-podinfo\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.791928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.774475 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3ce13d-da21-4593-8eba-603813119eea-istiod-ca-cert\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.791928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.744756 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-envoy\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.791928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.775315 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-certs\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.791928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.779207 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-data\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.791928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.780495 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gkr\" (UniqueName: \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-kube-api-access-m2gkr\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.809928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.792648 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:19.809928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.797259 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-istio-token\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:19.834928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.831095 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl"] Oct 29 22:22:19.834928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.831353 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" containerID="cri-o://cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d" gracePeriod=30 Oct 29 22:22:19.862933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:19.860170 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-envoy\") pod \"e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:20.149488 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:20.149443 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:20.492517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:20.489986 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:20.536668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:20.536640 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm"] Oct 29 22:22:20.540395 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:22:20.540369 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3ce13d_da21_4593_8eba_603813119eea.slice/crio-551a486cb13665e46d77f14b7361ec5d25daba09fbcb2da5ca9608a8bbd5be71 WatchSource:0}: Error finding container 551a486cb13665e46d77f14b7361ec5d25daba09fbcb2da5ca9608a8bbd5be71: Status 404 returned error can't find the container with id 551a486cb13665e46d77f14b7361ec5d25daba09fbcb2da5ca9608a8bbd5be71 Oct 29 22:22:20.544487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:20.544451 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:22:20.544581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:20.544527 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:22:20.544581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:20.544554 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:22:20.566416 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:20.566390 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" event={"ID":"3d3ce13d-da21-4593-8eba-603813119eea","Type":"ContainerStarted","Data":"551a486cb13665e46d77f14b7361ec5d25daba09fbcb2da5ca9608a8bbd5be71"} Oct 29 22:22:20.873335 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:20.873294 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:21.057925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:21.057529 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:21.510345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:21.508849 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:21.742375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:21.742330 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" event={"ID":"3d3ce13d-da21-4593-8eba-603813119eea","Type":"ContainerStarted","Data":"0f4c34fee3479fd25afe0124eedac71bf242f95d818452afd16ccd1a843cbacb"} Oct 29 22:22:21.813071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:21.812965 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" podStartSLOduration=2.81294598 podStartE2EDuration="2.81294598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:21.807291091 +0000 UTC m=+2407.281879543" watchObservedRunningTime="2026-10-29 22:22:21.81294598 +0000 UTC m=+2407.287534398" Oct 29 22:22:22.150531 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:22.150495 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:22.158713 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:22.158683 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:22.753635 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:22.753603 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:22.758439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:22.758408 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:22:22.892981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:22.892945 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:23.073773 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:23.073641 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:23.259819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:23.259773 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:24.697670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:24.697630 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:24.942518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:24.942471 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:25.329008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.328955 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:25.518765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.518732 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 22:22:25.705177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.705107 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-credential-socket\") pod \"4757bc87-b9b1-4f5a-944c-491770f8964c\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " Oct 29 22:22:25.705610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.705298 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-credential-socket" pod "4757bc87-b9b1-4f5a-944c-491770f8964c" (UID: "4757bc87-b9b1-4f5a-944c-491770f8964c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:25.705716 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.705690 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-data" pod "4757bc87-b9b1-4f5a-944c-491770f8964c" (UID: "4757bc87-b9b1-4f5a-944c-491770f8964c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:25.705872 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.705850 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-data\") pod \"4757bc87-b9b1-4f5a-944c-491770f8964c\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " Oct 29 22:22:25.705992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.705977 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-token\" (UniqueName: \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-token\") pod \"4757bc87-b9b1-4f5a-944c-491770f8964c\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " Oct 29 22:22:25.706034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.706018 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-kube-api-access-2zr6d\" (UniqueName: \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-kube-api-access-2zr6d\") pod \"4757bc87-b9b1-4f5a-944c-491770f8964c\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " Oct 29 22:22:25.706242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.706223 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4757bc87-b9b1-4f5a-944c-491770f8964c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4757bc87-b9b1-4f5a-944c-491770f8964c-istiod-ca-cert\") pod \"4757bc87-b9b1-4f5a-944c-491770f8964c\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " Oct 29 22:22:25.706411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.706396 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-socket\") pod \"4757bc87-b9b1-4f5a-944c-491770f8964c\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " Oct 29 22:22:25.706512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.706499 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-certs\") pod \"4757bc87-b9b1-4f5a-944c-491770f8964c\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " Oct 29 22:22:25.706571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.706532 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-envoy\") pod \"4757bc87-b9b1-4f5a-944c-491770f8964c\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " Oct 29 22:22:25.706626 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.706578 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-podinfo\") pod \"4757bc87-b9b1-4f5a-944c-491770f8964c\" (UID: \"4757bc87-b9b1-4f5a-944c-491770f8964c\") " Oct 29 22:22:25.706683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.706629 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4757bc87-b9b1-4f5a-944c-491770f8964c-istiod-ca-cert" pod "4757bc87-b9b1-4f5a-944c-491770f8964c" (UID: "4757bc87-b9b1-4f5a-944c-491770f8964c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:22:25.706683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.706645 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-socket" pod "4757bc87-b9b1-4f5a-944c-491770f8964c" (UID: "4757bc87-b9b1-4f5a-944c-491770f8964c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:25.707229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.706951 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-certs" pod "4757bc87-b9b1-4f5a-944c-491770f8964c" (UID: "4757bc87-b9b1-4f5a-944c-491770f8964c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:25.707426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.707401 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4757bc87-b9b1-4f5a-944c-491770f8964c-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.707564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.707431 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.707564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.707445 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.707564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.707459 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.707564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.707472 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.708858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.708817 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-token" pod "4757bc87-b9b1-4f5a-944c-491770f8964c" (UID: "4757bc87-b9b1-4f5a-944c-491770f8964c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:25.709438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.709314 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-kube-api-access-2zr6d" pod "4757bc87-b9b1-4f5a-944c-491770f8964c" (UID: "4757bc87-b9b1-4f5a-944c-491770f8964c"). InnerVolumeSpecName "kube-api-access-2zr6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:25.710271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.710248 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-podinfo" pod "4757bc87-b9b1-4f5a-944c-491770f8964c" (UID: "4757bc87-b9b1-4f5a-944c-491770f8964c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:22:25.717302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.717272 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-envoy" pod "4757bc87-b9b1-4f5a-944c-491770f8964c" (UID: "4757bc87-b9b1-4f5a-944c-491770f8964c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:25.780172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.780134 4952 generic.go:356] "Generic (PLEG): container finished" podID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerID="cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d" exitCode=0 Oct 29 22:22:25.780342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.780185 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" Oct 29 22:22:25.780342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.780215 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" event={"ID":"4757bc87-b9b1-4f5a-944c-491770f8964c","Type":"ContainerDied","Data":"cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d"} Oct 29 22:22:25.780342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.780330 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl" event={"ID":"4757bc87-b9b1-4f5a-944c-491770f8964c","Type":"ContainerDied","Data":"e719c94b08f1229d2aaf83016c2067c55f9f9347cb088ec23621cc25842727f1"} Oct 29 22:22:25.780563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.780361 4952 scope.go:122] "RemoveContainer" containerID="cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d" Oct 29 22:22:25.793788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.793770 4952 scope.go:122] "RemoveContainer" containerID="cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d" Oct 29 22:22:25.794125 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:22:25.794101 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d\": container with ID starting with cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d not found: ID does not exist" containerID="cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d" Oct 29 22:22:25.794214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.794135 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d"} err="failed to get container status \"cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d\": rpc error: code = NotFound desc = could not find container \"cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d\": container with ID starting with cfade3469e13552dd5f80a7886caf2993fdb87d6693c873826810d44afa1f84d not found: ID does not exist" Oct 29 22:22:25.809208 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.809183 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.809208 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.809209 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.809410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.809241 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.809410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.809319 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zr6d\" (UniqueName: \"kubernetes.io/projected/4757bc87-b9b1-4f5a-944c-491770f8964c-kube-api-access-2zr6d\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:25.852544 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.852503 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl"] Oct 29 22:22:25.852740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:25.852565 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-qt6wl"] Oct 29 22:22:26.660355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:26.659544 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" path="/var/lib/kubelet/pods/4757bc87-b9b1-4f5a-944c-491770f8964c/volumes" Oct 29 22:22:26.931670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:26.931575 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:27.105847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:27.105814 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:28.706062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:28.706015 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:29.035802 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:29.035717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:29.212005 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:29.211964 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:30.423280 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:30.423233 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:32.319101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.317105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:32.552638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.552595 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" podUID="9675baec-0ef5-4f3b-a9a0-be24996fb45f" containerName="main" containerID="cri-o://e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b" gracePeriod=45 Oct 29 22:22:32.855921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.855886 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:22:32.958216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.958182 4952 generic.go:356] "Generic (PLEG): container finished" podID="9675baec-0ef5-4f3b-a9a0-be24996fb45f" containerID="e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b" exitCode=0 Oct 29 22:22:32.958386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.958223 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" Oct 29 22:22:32.958386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.958262 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" event={"ID":"9675baec-0ef5-4f3b-a9a0-be24996fb45f","Type":"ContainerDied","Data":"e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b"} Oct 29 22:22:32.958386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.958307 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t" event={"ID":"9675baec-0ef5-4f3b-a9a0-be24996fb45f","Type":"ContainerDied","Data":"2ed3d926f78e1163a6eaaa80859e27748ab6d21b262ab7f4cc523df9ac5e1156"} Oct 29 22:22:32.958386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.958325 4952 scope.go:122] "RemoveContainer" containerID="e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b" Oct 29 22:22:32.971282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.971256 4952 scope.go:122] "RemoveContainer" containerID="e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b" Oct 29 22:22:32.971666 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:22:32.971631 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b\": container with ID starting with e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b not found: ID does not exist" containerID="e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b" Oct 29 22:22:32.971775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.971681 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b"} err="failed to get container status \"e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b\": rpc error: code = NotFound desc = could not find container \"e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b\": container with ID starting with e7ff540214c68b882e0892b5d673882ebfbeff1bf0ec3b1debc92d499e58093b not found: ID does not exist" Oct 29 22:22:32.988956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.988923 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tmp-dir\") pod \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " Oct 29 22:22:32.989114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.989031 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tls-certs\" (UniqueName: \"kubernetes.io/secret/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tls-certs\") pod \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " Oct 29 22:22:32.989114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.989098 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9675baec-0ef5-4f3b-a9a0-be24996fb45f-kube-api-access-92csv\" (UniqueName: \"kubernetes.io/projected/9675baec-0ef5-4f3b-a9a0-be24996fb45f-kube-api-access-92csv\") pod \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " Oct 29 22:22:32.989243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.989172 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-dshm\") pod \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " Oct 29 22:22:32.989302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.989166 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tmp-dir" pod "9675baec-0ef5-4f3b-a9a0-be24996fb45f" (UID: "9675baec-0ef5-4f3b-a9a0-be24996fb45f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:32.989398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.989360 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-home\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-home\") pod \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " Oct 29 22:22:32.989458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.989415 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-model-cache\") pod \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\" (UID: \"9675baec-0ef5-4f3b-a9a0-be24996fb45f\") " Oct 29 22:22:32.989765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.989694 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-home" pod "9675baec-0ef5-4f3b-a9a0-be24996fb45f" (UID: "9675baec-0ef5-4f3b-a9a0-be24996fb45f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:32.989765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.989728 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-model-cache" pod "9675baec-0ef5-4f3b-a9a0-be24996fb45f" (UID: "9675baec-0ef5-4f3b-a9a0-be24996fb45f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:32.990157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.990129 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:32.990272 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.990159 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:32.990272 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.990174 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:32.991609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.991578 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tls-certs" pod "9675baec-0ef5-4f3b-a9a0-be24996fb45f" (UID: "9675baec-0ef5-4f3b-a9a0-be24996fb45f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:22:32.991722 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.991581 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-dshm" pod "9675baec-0ef5-4f3b-a9a0-be24996fb45f" (UID: "9675baec-0ef5-4f3b-a9a0-be24996fb45f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:22:32.992042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:32.992024 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9675baec-0ef5-4f3b-a9a0-be24996fb45f-kube-api-access-92csv" pod "9675baec-0ef5-4f3b-a9a0-be24996fb45f" (UID: "9675baec-0ef5-4f3b-a9a0-be24996fb45f"). InnerVolumeSpecName "kube-api-access-92csv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:22:33.091495 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:33.091418 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9675baec-0ef5-4f3b-a9a0-be24996fb45f-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:33.091495 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:33.091451 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9675baec-0ef5-4f3b-a9a0-be24996fb45f-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:33.091495 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:33.091463 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92csv\" (UniqueName: \"kubernetes.io/projected/9675baec-0ef5-4f3b-a9a0-be24996fb45f-kube-api-access-92csv\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:22:33.311327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:33.311287 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t"] Oct 29 22:22:33.330250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:33.330204 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-675bf4/auth-test-model-675bf4-kserve-59b49f5b5-7gl5t"] Oct 29 22:22:34.037928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:34.036077 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:34.648031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:34.647987 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9675baec-0ef5-4f3b-a9a0-be24996fb45f" path="/var/lib/kubelet/pods/9675baec-0ef5-4f3b-a9a0-be24996fb45f/volumes" Oct 29 22:22:35.624973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:35.624930 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:36.493670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:36.493635 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:36.669136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:36.669098 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:36.968661 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:36.968624 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:37.389189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:37.389154 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:37.772552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:37.772423 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:38.039925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.039797 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:38.415428 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.415395 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv"] Oct 29 22:22:38.416243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.416217 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9675baec-0ef5-4f3b-a9a0-be24996fb45f" containerName="main" Oct 29 22:22:38.416243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.416243 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9675baec-0ef5-4f3b-a9a0-be24996fb45f" containerName="main" Oct 29 22:22:38.416432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.416261 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" Oct 29 22:22:38.416432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.416269 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" Oct 29 22:22:38.416534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.416506 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9675baec-0ef5-4f3b-a9a0-be24996fb45f" containerName="main" Oct 29 22:22:38.416534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.416521 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4757bc87-b9b1-4f5a-944c-491770f8964c" containerName="istio-proxy" Oct 29 22:22:38.422011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.421970 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.438831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.438799 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv"] Oct 29 22:22:38.455718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.455691 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-adfb9d-openshift-default-dockercfg-gjgqt\"" Oct 29 22:22:38.458832 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.458804 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fec7422-4663-4796-a686-a13c2d8c9d9f-istiod-ca-cert\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.459096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.459073 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-token\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.459226 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.459209 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-podinfo\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.459349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.459333 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-credential-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.459658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.459628 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-envoy\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.459857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.459837 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.463596 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.463571 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2tl6\" (UniqueName: \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-kube-api-access-m2tl6\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.463870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.463848 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-data\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.464122 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.464101 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-certs\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.500889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.500851 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:38.565720 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.565682 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-certs\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.565969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.565757 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fec7422-4663-4796-a686-a13c2d8c9d9f-istiod-ca-cert\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.565969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.565811 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-token\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.565969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.565834 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-podinfo\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.565969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.565857 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-credential-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.566238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.565980 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-envoy\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.566238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.566012 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.566238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.566032 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2tl6\" (UniqueName: \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-kube-api-access-m2tl6\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.566238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.566096 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-data\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.566238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.566165 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-certs\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.566504 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.566359 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-data\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.566504 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.566445 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-credential-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.567021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.566975 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.567210 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.567178 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fec7422-4663-4796-a686-a13c2d8c9d9f-istiod-ca-cert\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.568760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.568732 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-envoy\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.569367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.569340 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-podinfo\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.640942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.627960 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-token\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.640942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.636799 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2tl6\" (UniqueName: \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-kube-api-access-m2tl6\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:38.688979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.686561 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:38.765930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:38.763515 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:39.566548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:39.566515 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv"] Oct 29 22:22:39.567852 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:22:39.567820 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fec7422_4663_4796_a686_a13c2d8c9d9f.slice/crio-a9b44d4f68a5ae6100d888f79b43fc96314642e99f8e8acc7963e6bafd192b6a WatchSource:0}: Error finding container a9b44d4f68a5ae6100d888f79b43fc96314642e99f8e8acc7963e6bafd192b6a: Status 404 returned error can't find the container with id a9b44d4f68a5ae6100d888f79b43fc96314642e99f8e8acc7963e6bafd192b6a Oct 29 22:22:39.571760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:39.571701 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:22:39.571973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:39.571811 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:22:39.571973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:39.571855 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:22:39.931930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:39.929159 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:40.143985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:40.143943 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" event={"ID":"2fec7422-4663-4796-a686-a13c2d8c9d9f","Type":"ContainerStarted","Data":"023370a0306f54b1b66b96fd7602b766489c04ae2e90a0ca772558b55909954c"} Oct 29 22:22:40.144188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:40.144013 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" event={"ID":"2fec7422-4663-4796-a686-a13c2d8c9d9f","Type":"ContainerStarted","Data":"a9b44d4f68a5ae6100d888f79b43fc96314642e99f8e8acc7963e6bafd192b6a"} Oct 29 22:22:40.210685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:40.210562 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podStartSLOduration=2.21054518 podStartE2EDuration="2.21054518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:40.205669015 +0000 UTC m=+2425.680257435" watchObservedRunningTime="2026-10-29 22:22:40.21054518 +0000 UTC m=+2425.685133597" Oct 29 22:22:40.764464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:40.764424 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:40.773024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:40.772993 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:41.154109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:41.154074 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:41.155175 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:41.155148 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:22:41.815930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:41.815355 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:42.653157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:42.653108 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:42.833433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:42.833396 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:43.673960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:43.671682 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:44.404813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:44.404762 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:44.578509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:44.578472 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:44.971437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:44.971373 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z"] Oct 29 22:22:44.977292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:44.977258 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z"] Oct 29 22:22:44.977624 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:44.977601 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:22:45.007813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.007785 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-adfb9d\"" Oct 29 22:22:45.008011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.007827 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-adfb9d-dockercfg-xdh6r\"" Oct 29 22:22:45.050961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.050927 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4zh9\" (UniqueName: \"kubernetes.io/projected/ffe5ea27-066f-4f88-9f21-e5e09d289018-kube-api-access-r4zh9\") pod \"payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z\" (UID: \"ffe5ea27-066f-4f88-9f21-e5e09d289018\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:22:45.051165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.051085 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe5ea27-066f-4f88-9f21-e5e09d289018-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z\" (UID: \"ffe5ea27-066f-4f88-9f21-e5e09d289018\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:22:45.154262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.154223 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4zh9\" (UniqueName: \"kubernetes.io/projected/ffe5ea27-066f-4f88-9f21-e5e09d289018-kube-api-access-r4zh9\") pod \"payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z\" (UID: \"ffe5ea27-066f-4f88-9f21-e5e09d289018\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:22:45.154497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.154342 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe5ea27-066f-4f88-9f21-e5e09d289018-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z\" (UID: \"ffe5ea27-066f-4f88-9f21-e5e09d289018\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:22:45.155123 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.155101 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe5ea27-066f-4f88-9f21-e5e09d289018-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z\" (UID: \"ffe5ea27-066f-4f88-9f21-e5e09d289018\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:22:45.179854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.179819 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4zh9\" (UniqueName: \"kubernetes.io/projected/ffe5ea27-066f-4f88-9f21-e5e09d289018-kube-api-access-r4zh9\") pod \"payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z\" (UID: \"ffe5ea27-066f-4f88-9f21-e5e09d289018\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:22:45.300151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.300062 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:22:45.554982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.554245 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz"] Oct 29 22:22:45.591302 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:22:45.591034 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:22:45.753228 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.753186 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:45.823059 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.823018 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z"] Oct 29 22:22:45.861930 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:22:45.854665 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe5ea27_066f_4f88_9f21_e5e09d289018.slice/crio-16fea3efcf08cd380130e00fd24359774e0036b3d00356a13ac4b87befb4b724 WatchSource:0}: Error finding container 16fea3efcf08cd380130e00fd24359774e0036b3d00356a13ac4b87befb4b724: Status 404 returned error can't find the container with id 16fea3efcf08cd380130e00fd24359774e0036b3d00356a13ac4b87befb4b724 Oct 29 22:22:45.952297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:45.952201 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:46.212054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:46.212009 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" event={"ID":"ffe5ea27-066f-4f88-9f21-e5e09d289018","Type":"ContainerStarted","Data":"df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358"} Oct 29 22:22:46.212256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:46.212084 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:22:46.212256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:46.212102 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" event={"ID":"ffe5ea27-066f-4f88-9f21-e5e09d289018","Type":"ContainerStarted","Data":"16fea3efcf08cd380130e00fd24359774e0036b3d00356a13ac4b87befb4b724"} Oct 29 22:22:46.262048 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:46.261984 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" podStartSLOduration=2.261966966 podStartE2EDuration="2.261966966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:46.255427545 +0000 UTC m=+2431.730015964" watchObservedRunningTime="2026-10-29 22:22:46.261966966 +0000 UTC m=+2431.736555385" Oct 29 22:22:47.753498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:47.752950 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:49.557933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:49.551655 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:51.134333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:51.134287 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:52.662232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:52.661629 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:54.352890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:54.352844 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:55.838060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:55.838023 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:56.045567 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:56.045529 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:56.480706 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:56.480664 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:56.991869 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:56.991824 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:57.242800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:57.242710 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:57.251128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:57.251096 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:22:57.930558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:57.930475 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:59.416835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:59.416798 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:59.594867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:59.594824 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:22:59.600986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:59.600899 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:59.806047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:22:59.805954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:00.622346 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:00.622304 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" podUID="465af292-e674-41e0-9047-733f7fdf93ce" containerName="main" containerID="cri-o://9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc" gracePeriod=45 Oct 29 22:23:01.305937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.303531 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:23:01.368838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.364451 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-dshm\") pod \"465af292-e674-41e0-9047-733f7fdf93ce\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " Oct 29 22:23:01.368838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.364534 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-model-cache\") pod \"465af292-e674-41e0-9047-733f7fdf93ce\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " Oct 29 22:23:01.368838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.364714 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-home\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-home\") pod \"465af292-e674-41e0-9047-733f7fdf93ce\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " Oct 29 22:23:01.368838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.364782 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465af292-e674-41e0-9047-733f7fdf93ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/465af292-e674-41e0-9047-733f7fdf93ce-tls-certs\") pod \"465af292-e674-41e0-9047-733f7fdf93ce\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " Oct 29 22:23:01.368838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.364871 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465af292-e674-41e0-9047-733f7fdf93ce-kube-api-access-9msll\" (UniqueName: \"kubernetes.io/projected/465af292-e674-41e0-9047-733f7fdf93ce-kube-api-access-9msll\") pod \"465af292-e674-41e0-9047-733f7fdf93ce\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " Oct 29 22:23:01.368838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.365000 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-tmp-dir\") pod \"465af292-e674-41e0-9047-733f7fdf93ce\" (UID: \"465af292-e674-41e0-9047-733f7fdf93ce\") " Oct 29 22:23:01.368838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.366127 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-tmp-dir" pod "465af292-e674-41e0-9047-733f7fdf93ce" (UID: "465af292-e674-41e0-9047-733f7fdf93ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:01.368838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.368146 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-home" pod "465af292-e674-41e0-9047-733f7fdf93ce" (UID: "465af292-e674-41e0-9047-733f7fdf93ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:01.368838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.368291 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-model-cache" pod "465af292-e674-41e0-9047-733f7fdf93ce" (UID: "465af292-e674-41e0-9047-733f7fdf93ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:01.445372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.445309 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465af292-e674-41e0-9047-733f7fdf93ce-tls-certs" pod "465af292-e674-41e0-9047-733f7fdf93ce" (UID: "465af292-e674-41e0-9047-733f7fdf93ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:23:01.461190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.447144 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-dshm" pod "465af292-e674-41e0-9047-733f7fdf93ce" (UID: "465af292-e674-41e0-9047-733f7fdf93ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:01.461190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.449894 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465af292-e674-41e0-9047-733f7fdf93ce-kube-api-access-9msll" pod "465af292-e674-41e0-9047-733f7fdf93ce" (UID: "465af292-e674-41e0-9047-733f7fdf93ce"). InnerVolumeSpecName "kube-api-access-9msll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:01.469061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.467514 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:23:01.469061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.467551 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465af292-e674-41e0-9047-733f7fdf93ce-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:23:01.469061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.467568 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msll\" (UniqueName: \"kubernetes.io/projected/465af292-e674-41e0-9047-733f7fdf93ce-kube-api-access-9msll\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:23:01.469061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.467607 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:23:01.469061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.467623 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:23:01.469061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.467638 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465af292-e674-41e0-9047-733f7fdf93ce-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:23:01.645364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.640493 4952 generic.go:356] "Generic (PLEG): container finished" podID="465af292-e674-41e0-9047-733f7fdf93ce" containerID="9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc" exitCode=0 Oct 29 22:23:01.645364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.640698 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" event={"ID":"465af292-e674-41e0-9047-733f7fdf93ce","Type":"ContainerDied","Data":"9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc"} Oct 29 22:23:01.645364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.640738 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" event={"ID":"465af292-e674-41e0-9047-733f7fdf93ce","Type":"ContainerDied","Data":"e8ed837e527f62b132d66a7cc34c00e9cea5ba0ed93ba95d586bdacdc052e1b1"} Oct 29 22:23:01.645364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.640760 4952 scope.go:122] "RemoveContainer" containerID="9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc" Oct 29 22:23:01.645364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.640938 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" Oct 29 22:23:01.687267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.687154 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:01.700648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.698068 4952 scope.go:122] "RemoveContainer" containerID="9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc" Oct 29 22:23:01.700648 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:23:01.699267 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc\": container with ID starting with 9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc not found: ID does not exist" containerID="9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc" Oct 29 22:23:01.700648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.699302 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc"} err="failed to get container status \"9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc\": rpc error: code = NotFound desc = could not find container \"9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc\": container with ID starting with 9db8ef9e7bdd82a479622ef433ea81692ffd97e05d64376175b010ab330c07bc not found: ID does not exist" Oct 29 22:23:01.744312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.742965 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz"] Oct 29 22:23:01.746875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.746846 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz"] Oct 29 22:23:01.868963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.867694 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:01.982929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:01.980015 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-1e74c3/auth-test-model-1e74c3-kserve-855fddffbf-kzgvz" podUID="465af292-e674-41e0-9047-733f7fdf93ce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.148:8000/ready\": context deadline exceeded" Oct 29 22:23:02.651925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:02.650496 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465af292-e674-41e0-9047-733f7fdf93ce" path="/var/lib/kubelet/pods/465af292-e674-41e0-9047-733f7fdf93ce/volumes" Oct 29 22:23:02.754983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:02.753189 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:03.445135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:03.445093 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:03.639979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:03.639889 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:04.005127 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.005095 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s"] Oct 29 22:23:04.017726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.017694 4952 cpu_manager.go:414] "containerMap: removing container" podUID="465af292-e674-41e0-9047-733f7fdf93ce" containerName="main" Oct 29 22:23:04.017976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.017957 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465af292-e674-41e0-9047-733f7fdf93ce" containerName="main" Oct 29 22:23:04.018649 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.018622 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="465af292-e674-41e0-9047-733f7fdf93ce" containerName="main" Oct 29 22:23:04.025124 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.025096 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s"] Oct 29 22:23:04.025311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.025205 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:23:04.043819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.043787 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-529e60-dockercfg-5m42k\"" Oct 29 22:23:04.057241 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.057210 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-529e60\"" Oct 29 22:23:04.105703 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.105656 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-plugins-config-volume\") pod \"payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s\" (UID: \"99bc6aed-e198-4f9f-8a2b-54eaaea369ec\") " pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:23:04.105979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.105946 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vbfv\" (UniqueName: \"kubernetes.io/projected/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-kube-api-access-8vbfv\") pod \"payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s\" (UID: \"99bc6aed-e198-4f9f-8a2b-54eaaea369ec\") " pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:23:04.207625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.207591 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-plugins-config-volume\") pod \"payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s\" (UID: \"99bc6aed-e198-4f9f-8a2b-54eaaea369ec\") " pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:23:04.207817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.207801 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vbfv\" (UniqueName: \"kubernetes.io/projected/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-kube-api-access-8vbfv\") pod \"payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s\" (UID: \"99bc6aed-e198-4f9f-8a2b-54eaaea369ec\") " pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:23:04.208356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.208333 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-plugins-config-volume\") pod \"payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s\" (UID: \"99bc6aed-e198-4f9f-8a2b-54eaaea369ec\") " pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:23:04.234569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.234537 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vbfv\" (UniqueName: \"kubernetes.io/projected/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-kube-api-access-8vbfv\") pod \"payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s\" (UID: \"99bc6aed-e198-4f9f-8a2b-54eaaea369ec\") " pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:23:04.346889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.346189 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:23:04.689628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.689599 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s"] Oct 29 22:23:04.747918 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.747868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:04.927564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.926216 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" event={"ID":"99bc6aed-e198-4f9f-8a2b-54eaaea369ec","Type":"ContainerStarted","Data":"7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365"} Oct 29 22:23:04.927564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.926286 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" event={"ID":"99bc6aed-e198-4f9f-8a2b-54eaaea369ec","Type":"ContainerStarted","Data":"40555f651f3666dfa8ef7940cd8c959ef3514dfd66c1bf6c65e4dbf4d5f42310"} Oct 29 22:23:04.927564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.927346 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:23:04.976024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:04.975959 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" podStartSLOduration=1.975942174 podStartE2EDuration="1.975942174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:04.971606145 +0000 UTC m=+2450.446194565" watchObservedRunningTime="2026-10-29 22:23:04.975942174 +0000 UTC m=+2450.450530590" Oct 29 22:23:06.572862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:06.572825 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:06.755677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:06.755634 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:06.986060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:06.986023 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:08.550503 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:08.550467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:10.145455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:10.145415 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:10.490725 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:10.490618 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:11.806977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:11.806930 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:13.410926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:13.407342 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:14.948556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:14.948517 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm"] Oct 29 22:23:14.953178 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:14.953155 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:14.975039 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:14.975009 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm"] Oct 29 22:23:14.975886 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:14.975861 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-4c1015-serving-cert\"" Oct 29 22:23:15.088783 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.088744 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:15.096781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.096751 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.097008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.096870 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbkw\" (UniqueName: \"kubernetes.io/projected/9589038c-da0d-4e10-8a0c-05d93e455357-kube-api-access-4tbkw\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.097008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.096983 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-metrics-certs\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.097156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.097107 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9589038c-da0d-4e10-8a0c-05d93e455357-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.200457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.200360 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.200788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.200761 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbkw\" (UniqueName: \"kubernetes.io/projected/9589038c-da0d-4e10-8a0c-05d93e455357-kube-api-access-4tbkw\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.200992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.200974 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-metrics-certs\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.201243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.201221 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9589038c-da0d-4e10-8a0c-05d93e455357-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.202295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.202270 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9589038c-da0d-4e10-8a0c-05d93e455357-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.211838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.211772 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.223474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.223447 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-metrics-certs\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.233149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.233105 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbkw\" (UniqueName: \"kubernetes.io/projected/9589038c-da0d-4e10-8a0c-05d93e455357-kube-api-access-4tbkw\") pod \"maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.257800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.257766 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:15.268807 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.268778 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:15.477532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.477441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:15.531280 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:23:15.531243 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:23:15.589326 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.589298 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm"] Oct 29 22:23:15.589471 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:23:15.589389 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9589038c_da0d_4e10_8a0c_05d93e455357.slice/crio-0a92c78bb214e8b4fe0f0e06ec3da73e6b64dbf237bc8863b12854e6b7945e7e WatchSource:0}: Error finding container 0a92c78bb214e8b4fe0f0e06ec3da73e6b64dbf237bc8863b12854e6b7945e7e: Status 404 returned error can't find the container with id 0a92c78bb214e8b4fe0f0e06ec3da73e6b64dbf237bc8863b12854e6b7945e7e Oct 29 22:23:15.590826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:15.590807 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:23:16.293212 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.292973 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" event={"ID":"9589038c-da0d-4e10-8a0c-05d93e455357","Type":"ContainerStarted","Data":"6ef1ba1ef02280c6a2133c686e558a43ff8c148cab372a78867293aa5c2f8a08"} Oct 29 22:23:16.293212 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.293067 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" event={"ID":"9589038c-da0d-4e10-8a0c-05d93e455357","Type":"ContainerStarted","Data":"0a92c78bb214e8b4fe0f0e06ec3da73e6b64dbf237bc8863b12854e6b7945e7e"} Oct 29 22:23:16.293212 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.293137 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:16.339408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.339337 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" podStartSLOduration=1.9636432080000001 podStartE2EDuration="2.33931772s" totalImagesPullingTime="375.674512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:16.338132858 +0000 UTC m=+2461.812721277" watchObservedRunningTime="2026-10-29 22:23:16.33931772 +0000 UTC m=+2461.813906140" Oct 29 22:23:16.834781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.834744 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h"] Oct 29 22:23:16.839388 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.839370 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:16.861660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.861546 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-50d4af-serving-cert\"" Oct 29 22:23:16.867645 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.867607 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h"] Oct 29 22:23:16.970233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.970198 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-metrics-certs\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:16.970427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.970247 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:16.970427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.970346 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-maas-api-tls\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:16.970614 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.970590 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvr7\" (UniqueName: \"kubernetes.io/projected/8b8892a7-74e6-422d-a0df-3d0ece183682-kube-api-access-jmvr7\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:16.970721 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.970702 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8892a7-74e6-422d-a0df-3d0ece183682-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:16.989378 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:16.989351 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:23:17.083008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.082849 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-maas-api-tls\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:17.083262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.083137 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvr7\" (UniqueName: \"kubernetes.io/projected/8b8892a7-74e6-422d-a0df-3d0ece183682-kube-api-access-jmvr7\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:17.083262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.083240 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8892a7-74e6-422d-a0df-3d0ece183682-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:17.083428 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.083411 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-metrics-certs\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:17.091788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.091083 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8892a7-74e6-422d-a0df-3d0ece183682-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:17.110786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.110613 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-maas-api-tls\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:17.119707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.119587 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-metrics-certs\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:17.149649 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.149617 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:17.156832 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.156798 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvr7\" (UniqueName: \"kubernetes.io/projected/8b8892a7-74e6-422d-a0df-3d0ece183682-kube-api-access-jmvr7\") pod \"maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:17.183552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.183313 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:17.533968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.533933 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h"] Oct 29 22:23:17.535039 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:23:17.535007 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8892a7_74e6_422d_a0df_3d0ece183682.slice/crio-45873fcdbba0467ff935ffe8f6121d78d5243bc5c0e777f88b646b54c9302803 WatchSource:0}: Error finding container 45873fcdbba0467ff935ffe8f6121d78d5243bc5c0e777f88b646b54c9302803: Status 404 returned error can't find the container with id 45873fcdbba0467ff935ffe8f6121d78d5243bc5c0e777f88b646b54c9302803 Oct 29 22:23:17.536873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.536845 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:17.772932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:17.772612 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:18.245009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:18.229738 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:18.431927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:18.428933 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" event={"ID":"8b8892a7-74e6-422d-a0df-3d0ece183682","Type":"ContainerStarted","Data":"45873fcdbba0467ff935ffe8f6121d78d5243bc5c0e777f88b646b54c9302803"} Oct 29 22:23:19.442409 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:19.442368 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" event={"ID":"8b8892a7-74e6-422d-a0df-3d0ece183682","Type":"ContainerStarted","Data":"5040a654068cec765be98466a5e3376cd3fe83ec7f36297efdb7b7bd678302bf"} Oct 29 22:23:19.442930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:19.442431 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:19.491447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:19.491390 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" podStartSLOduration=2.913625164 podStartE2EDuration="3.491374015s" totalImagesPullingTime="577.748851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:19.491107587 +0000 UTC m=+2464.965696007" watchObservedRunningTime="2026-10-29 22:23:19.491374015 +0000 UTC m=+2464.965962433" Oct 29 22:23:19.635950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:19.635838 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:21.199588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:21.199549 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:22.311216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:22.311185 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:23:22.839660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:22.839612 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:24.431536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:24.431494 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:25.368894 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:25.368853 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:25.455551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:25.455518 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:23:25.542442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:25.542402 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:25.727129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:25.727040 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:26.123551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:26.123495 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:27.664925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:27.651624 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:29.336954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:29.336891 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:30.948553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:30.948516 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:32.661271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:32.661207 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:32.853199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:32.853159 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:33.794371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:33.794330 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:33.994455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:33.994415 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:34.728515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:34.728478 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:35.405417 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:35.405382 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:36.572671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:36.572628 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:36.750230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:36.750191 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:36.950982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:36.950899 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:38.349963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:38.349925 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:40.025021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:40.024981 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:41.723951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:41.720946 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:43.401298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:43.401240 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:45.019044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:45.018964 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:45.532348 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:23:45.532307 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:23:46.695633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:46.695595 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:48.397534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:48.397494 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:50.056153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:50.051517 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:51.754952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:51.746503 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:53.717042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:53.717003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:55.532825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:55.532786 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:57.353762 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:57.329348 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:57.502551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:57.502519 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:57.681230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:57.681130 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:58.924930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:23:58.921389 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:00.847591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:00.847463 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:02.646020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:02.642130 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:04.230745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:04.230633 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:05.717206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:05.717097 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:06.149927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:06.147437 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:06.370296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:06.370261 4952 generic.go:356] "Generic (PLEG): container finished" podID="3d3ce13d-da21-4593-8eba-603813119eea" containerID="0f4c34fee3479fd25afe0124eedac71bf242f95d818452afd16ccd1a843cbacb" exitCode=0 Oct 29 22:24:06.370515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:06.370314 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" event={"ID":"3d3ce13d-da21-4593-8eba-603813119eea","Type":"ContainerDied","Data":"0f4c34fee3479fd25afe0124eedac71bf242f95d818452afd16ccd1a843cbacb"} Oct 29 22:24:06.370965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:06.370885 4952 scope.go:122] "RemoveContainer" containerID="0f4c34fee3479fd25afe0124eedac71bf242f95d818452afd16ccd1a843cbacb" Oct 29 22:24:06.373352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:06.373318 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:06.373463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:06.373395 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:06.373463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:06.373441 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:07.381964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:07.381927 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" event={"ID":"3d3ce13d-da21-4593-8eba-603813119eea","Type":"ContainerStarted","Data":"1b537c510231bde3cce31639dd7a1663adf9dd9dd0c04797fb48ceb093f731db"} Oct 29 22:24:07.755106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:07.755008 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:24:08.152046 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:08.151995 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:24:08.156094 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:08.156052 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:08.156690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:08.156664 4952 patch_prober.go:29] interesting pod/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:24:08.156975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:08.156941 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 22:24:09.150021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:09.149983 4952 patch_prober.go:29] interesting pod/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:24:09.150603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:09.150574 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 22:24:09.887932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:09.884273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:10.151981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:10.151238 4952 patch_prober.go:29] interesting pod/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:24:10.151981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:10.151311 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 22:24:10.632422 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:10.632387 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:11.153926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:11.153191 4952 patch_prober.go:29] interesting pod/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:24:11.153926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:11.153252 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 22:24:11.760716 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:11.760676 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:12.152927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:12.152213 4952 patch_prober.go:29] interesting pod/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:24:12.152927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:12.152274 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 22:24:13.158882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:13.158841 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:24:13.160278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:13.160248 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:24:13.531980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:13.523890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:15.294931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:15.293528 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:15.530043 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:24:15.530000 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:24:17.080523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:17.080434 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:18.763057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:18.763008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:20.424317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:20.424263 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:20.658799 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:20.658326 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:21.355285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:21.355249 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:21.524015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:21.523973 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:21.907556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:21.907515 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:22.088070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:22.088034 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:22.309187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:22.309105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:22.553784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:22.553696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:22.728533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:22.728502 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:23.336585 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:23.336547 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:23.513583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:23.513550 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:23.710990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:23.706375 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:24.661648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:24.661603 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:24:24.688708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:24.688651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:24.860846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:24.860809 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:25.066214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:25.066119 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:25.239501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:25.239464 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:27.325460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:27.325337 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:27.899099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:27.899054 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:28.065795 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:28.065755 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:29.124956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:29.124865 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:29.768511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:29.768471 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:24:29.828646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:29.828604 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:29.862416 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:29.862374 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8"] Oct 29 22:24:29.894929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:29.890277 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8"] Oct 29 22:24:29.894929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:29.890429 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:24:29.914392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:29.910102 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-797e0c-dockercfg-6fjw2\"" Oct 29 22:24:29.914392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:29.910191 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-797e0c\"" Oct 29 22:24:29.943567 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:29.943528 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4dd6\" (UniqueName: \"kubernetes.io/projected/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-kube-api-access-h4dd6\") pod \"payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8\" (UID: \"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:24:29.943755 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:29.943609 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8\" (UID: \"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:24:30.029985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:30.025028 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:30.103926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:30.071154 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4dd6\" (UniqueName: \"kubernetes.io/projected/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-kube-api-access-h4dd6\") pod \"payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8\" (UID: \"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:24:30.111929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:30.108031 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8\" (UID: \"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:24:30.144966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:30.116780 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8\" (UID: \"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:24:30.175865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:30.175565 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4dd6\" (UniqueName: \"kubernetes.io/projected/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-kube-api-access-h4dd6\") pod \"payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8\" (UID: \"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:24:30.213572 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:30.213262 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:24:30.662612 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:30.662577 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8"] Oct 29 22:24:30.907127 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:30.906992 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" event={"ID":"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358","Type":"ContainerStarted","Data":"feef140a7abefa0433a68b93e763da5ea0113e2ccc95f6c9a4c3278e3a4127c9"} Oct 29 22:24:31.296704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:31.296546 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:31.952932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:31.951316 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" event={"ID":"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358","Type":"ContainerStarted","Data":"8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e"} Oct 29 22:24:31.956929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:31.955920 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:24:32.006020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:32.005959 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" podStartSLOduration=3.005939432 podStartE2EDuration="3.005939432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:32.003395019 +0000 UTC m=+2537.477983438" watchObservedRunningTime="2026-10-29 22:24:32.005939432 +0000 UTC m=+2537.480527851" Oct 29 22:24:33.621548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:33.621503 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:35.735926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:35.732570 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:37.685280 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:37.678946 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:39.545751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:39.542664 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:40.650942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:40.648316 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:41.337643 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:41.337602 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:41.861110 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:41.861071 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:42.034387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:42.034347 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:42.201754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:42.201665 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:42.204061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:42.204031 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:24:42.204203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:42.204079 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:24:42.476373 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:42.476273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:42.656832 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:42.656801 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:42.968835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:42.968809 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:24:43.198047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:43.197638 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:43.233718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:43.233670 4952 generic.go:356] "Generic (PLEG): container finished" podID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerID="023370a0306f54b1b66b96fd7602b766489c04ae2e90a0ca772558b55909954c" exitCode=0 Oct 29 22:24:43.233894 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:43.233757 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" event={"ID":"2fec7422-4663-4796-a686-a13c2d8c9d9f","Type":"ContainerDied","Data":"023370a0306f54b1b66b96fd7602b766489c04ae2e90a0ca772558b55909954c"} Oct 29 22:24:43.234314 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:43.234293 4952 scope.go:122] "RemoveContainer" containerID="023370a0306f54b1b66b96fd7602b766489c04ae2e90a0ca772558b55909954c" Oct 29 22:24:43.236962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:43.236874 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:43.237070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:43.237028 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:43.237165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:43.237077 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:43.394964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:43.393792 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:44.247989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:44.247953 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" event={"ID":"2fec7422-4663-4796-a686-a13c2d8c9d9f","Type":"ContainerStarted","Data":"ef1202e90fe228756aeb958568f646f491ccbf4c56e1b3da57a1374a37a331d0"} Oct 29 22:24:44.765148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:44.765115 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:24:44.765400 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:44.765376 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:24:44.765460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:44.765429 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 22:24:45.040243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:45.040158 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:45.558959 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:24:45.558857 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:24:45.765244 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:45.765198 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:24:45.765457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:45.765270 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 22:24:46.481483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:46.481443 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:46.650795 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:46.650764 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:46.765428 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:46.765327 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:24:46.765732 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:46.765699 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 22:24:47.122486 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:47.122448 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7"] Oct 29 22:24:47.130076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:47.130037 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" containerName="istio-proxy" containerID="cri-o://1a0f0b2000f8f89c50c528196601bd0b906e0a5355278ecf058dfb7bf8f76311" gracePeriod=30 Oct 29 22:24:47.474987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:47.474878 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:47.654599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:47.654555 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:47.764966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:47.764856 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:24:47.765181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:47.765157 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 22:24:48.102528 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:48.102494 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:48.667559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:48.667513 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:48.769389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:48.769348 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:24:48.769709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:48.769673 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 22:24:48.844834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:48.844797 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:49.764983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:49.764950 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:24:49.765450 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:49.765425 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 22:24:50.184319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:50.184270 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:50.495986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:50.488539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:50.765034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:50.764932 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:24:50.765608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:50.765571 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 22:24:51.773047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:51.773007 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:24:51.774288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:51.774255 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:24:51.775625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:51.775602 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:24:52.065160 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.064999 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:52.425196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.425144 4952 generic.go:356] "Generic (PLEG): container finished" podID="4d700972-dd64-4e07-b92f-27fb31c21484" containerID="1a0f0b2000f8f89c50c528196601bd0b906e0a5355278ecf058dfb7bf8f76311" exitCode=0 Oct 29 22:24:52.426075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.426040 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" event={"ID":"4d700972-dd64-4e07-b92f-27fb31c21484","Type":"ContainerDied","Data":"1a0f0b2000f8f89c50c528196601bd0b906e0a5355278ecf058dfb7bf8f76311"} Oct 29 22:24:52.508378 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.508348 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:24:52.591852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.591814 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-envoy\") pod \"4d700972-dd64-4e07-b92f-27fb31c21484\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " Oct 29 22:24:52.592183 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.592153 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-kube-api-access-qbvlt\" (UniqueName: \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-kube-api-access-qbvlt\") pod \"4d700972-dd64-4e07-b92f-27fb31c21484\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " Oct 29 22:24:52.592396 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.592371 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-credential-socket\") pod \"4d700972-dd64-4e07-b92f-27fb31c21484\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " Oct 29 22:24:52.592527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.592408 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-certs\") pod \"4d700972-dd64-4e07-b92f-27fb31c21484\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " Oct 29 22:24:52.592527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.592464 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-socket\") pod \"4d700972-dd64-4e07-b92f-27fb31c21484\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " Oct 29 22:24:52.592689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.592558 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-data\") pod \"4d700972-dd64-4e07-b92f-27fb31c21484\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " Oct 29 22:24:52.592689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.592616 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d700972-dd64-4e07-b92f-27fb31c21484-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d700972-dd64-4e07-b92f-27fb31c21484-istio-podinfo\") pod \"4d700972-dd64-4e07-b92f-27fb31c21484\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " Oct 29 22:24:52.592689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.592644 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-credential-socket" pod "4d700972-dd64-4e07-b92f-27fb31c21484" (UID: "4d700972-dd64-4e07-b92f-27fb31c21484"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:24:52.592845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.592710 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d700972-dd64-4e07-b92f-27fb31c21484-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d700972-dd64-4e07-b92f-27fb31c21484-istiod-ca-cert\") pod \"4d700972-dd64-4e07-b92f-27fb31c21484\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " Oct 29 22:24:52.592845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.592778 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-istio-token\" (UniqueName: \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-istio-token\") pod \"4d700972-dd64-4e07-b92f-27fb31c21484\" (UID: \"4d700972-dd64-4e07-b92f-27fb31c21484\") " Oct 29 22:24:52.593002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.592859 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-socket" pod "4d700972-dd64-4e07-b92f-27fb31c21484" (UID: "4d700972-dd64-4e07-b92f-27fb31c21484"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:24:52.593344 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.593065 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-certs" pod "4d700972-dd64-4e07-b92f-27fb31c21484" (UID: "4d700972-dd64-4e07-b92f-27fb31c21484"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:24:52.593586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.593521 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:24:52.593586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.593542 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:24:52.593586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.593556 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:24:52.593789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.593600 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d700972-dd64-4e07-b92f-27fb31c21484-istiod-ca-cert" pod "4d700972-dd64-4e07-b92f-27fb31c21484" (UID: "4d700972-dd64-4e07-b92f-27fb31c21484"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:24:52.593866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.593809 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-data" pod "4d700972-dd64-4e07-b92f-27fb31c21484" (UID: "4d700972-dd64-4e07-b92f-27fb31c21484"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:24:52.595163 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.595133 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-envoy" pod "4d700972-dd64-4e07-b92f-27fb31c21484" (UID: "4d700972-dd64-4e07-b92f-27fb31c21484"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:24:52.596143 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.596119 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-istio-token" pod "4d700972-dd64-4e07-b92f-27fb31c21484" (UID: "4d700972-dd64-4e07-b92f-27fb31c21484"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:52.596427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.596402 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d700972-dd64-4e07-b92f-27fb31c21484-istio-podinfo" pod "4d700972-dd64-4e07-b92f-27fb31c21484" (UID: "4d700972-dd64-4e07-b92f-27fb31c21484"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:24:52.596877 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.596856 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-kube-api-access-qbvlt" pod "4d700972-dd64-4e07-b92f-27fb31c21484" (UID: "4d700972-dd64-4e07-b92f-27fb31c21484"). InnerVolumeSpecName "kube-api-access-qbvlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:52.695107 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.695065 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:24:52.695107 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.695107 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d700972-dd64-4e07-b92f-27fb31c21484-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:24:52.695329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.695129 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d700972-dd64-4e07-b92f-27fb31c21484-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:24:52.695329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.695143 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:24:52.695329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.695155 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d700972-dd64-4e07-b92f-27fb31c21484-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:24:52.695329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:52.695166 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbvlt\" (UniqueName: \"kubernetes.io/projected/4d700972-dd64-4e07-b92f-27fb31c21484-kube-api-access-qbvlt\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:24:53.436970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:53.436936 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" Oct 29 22:24:53.437434 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:53.436937 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7" event={"ID":"4d700972-dd64-4e07-b92f-27fb31c21484","Type":"ContainerDied","Data":"a8404671e2867828dfb6386243049d7270075e72e146b19f396c36b72c695d12"} Oct 29 22:24:53.437434 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:53.437088 4952 scope.go:122] "RemoveContainer" containerID="1a0f0b2000f8f89c50c528196601bd0b906e0a5355278ecf058dfb7bf8f76311" Oct 29 22:24:53.513305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:53.513269 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7"] Oct 29 22:24:53.529124 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:53.529084 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bef83b-openshift-default-7f7c79c759-hl8z7"] Oct 29 22:24:53.744984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:53.739120 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:54.667329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:54.667280 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" path="/var/lib/kubelet/pods/4d700972-dd64-4e07-b92f-27fb31c21484/volumes" Oct 29 22:24:54.834608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:54.834568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:55.013518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:55.013428 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:55.706519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:55.706488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:55.896643 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:55.896563 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:57.172014 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.171961 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r"] Oct 29 22:24:57.173186 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.173151 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" containerName="istio-proxy" Oct 29 22:24:57.173186 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.173182 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" containerName="istio-proxy" Oct 29 22:24:57.173463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.173446 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d700972-dd64-4e07-b92f-27fb31c21484" containerName="istio-proxy" Oct 29 22:24:57.178736 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.178713 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.208454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.208412 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-07e166-openshift-default-dockercfg-nf577\"" Oct 29 22:24:57.216141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.216097 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r"] Oct 29 22:24:57.281500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.281448 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-podinfo\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.281680 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.281642 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istiod-ca-cert\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.281740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.281693 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-token\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.281853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.281819 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtg4\" (UniqueName: \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-kube-api-access-xwtg4\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.281969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.281888 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-data\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.282049 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.282009 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-envoy\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.282113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.282061 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-credential-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.282172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.282155 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.282320 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.282291 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-certs\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.384170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384124 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-certs\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.384374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384241 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-podinfo\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.384374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384344 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istiod-ca-cert\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.384497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384388 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-token\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.384497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384417 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtg4\" (UniqueName: \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-kube-api-access-xwtg4\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.384497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384445 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-data\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.384666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384624 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-certs\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.384666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384642 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-envoy\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.385404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384711 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-credential-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.385404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384784 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.385404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.384860 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-data\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.385404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.385115 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-credential-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.385404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.385184 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-socket\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.385404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.385227 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istiod-ca-cert\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.387101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.387069 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-envoy\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.387274 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.387259 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-podinfo\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.436482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.436364 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-token\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.443395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.443260 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtg4\" (UniqueName: \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-kube-api-access-xwtg4\") pod \"e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.464021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.463529 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:57.495547 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.495517 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:57.835072 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.835021 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r"] Oct 29 22:24:57.836884 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:24:57.836853 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3543df_0b4d_4a83_ba52_ed177a1dfb93.slice/crio-64a7758aee6e464c48bda02d9d471027c2fe247fc2e8dd14f00a6e8552bb8148 WatchSource:0}: Error finding container 64a7758aee6e464c48bda02d9d471027c2fe247fc2e8dd14f00a6e8552bb8148: Status 404 returned error can't find the container with id 64a7758aee6e464c48bda02d9d471027c2fe247fc2e8dd14f00a6e8552bb8148 Oct 29 22:24:57.839967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.839922 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:57.840108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.840021 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:57.840108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.840064 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:57.841395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.841370 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5"] Oct 29 22:24:57.852748 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.852698 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:57.878042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.878007 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5"] Oct 29 22:24:57.901196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.901053 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:57.901196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.901115 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:57.901428 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.901408 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:57.901515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.901496 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlrhb\" (UniqueName: \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-kube-api-access-nlrhb\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:57.901636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.901618 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01b1e90-e2d0-4547-a491-f91d584b6253-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:57.901708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.901672 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:57.901804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.901775 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:57.901871 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.901823 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:57.901943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:57.901930 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.003452 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.003417 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.003681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.003477 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrhb\" (UniqueName: \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-kube-api-access-nlrhb\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.003681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.003532 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01b1e90-e2d0-4547-a491-f91d584b6253-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.003681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.003562 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.003681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.003589 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.003681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.003608 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.004025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.003697 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.004025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.003743 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.004025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.003782 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.004025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.003950 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.005009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.004884 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.005254 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.005231 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.005488 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.005466 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.005685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.005663 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01b1e90-e2d0-4547-a491-f91d584b6253-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.006749 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.006720 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.007423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.007406 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.030249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.030202 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.031237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.030454 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlrhb\" (UniqueName: \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-kube-api-access-nlrhb\") pod \"maas-default-gateway-openshift-default-6f54779779-xfcv5\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.195356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.195264 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:24:58.499591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.499484 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" event={"ID":"8e3543df-0b4d-4a83-ba52-ed177a1dfb93","Type":"ContainerStarted","Data":"225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838"} Oct 29 22:24:58.499591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.499561 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" event={"ID":"8e3543df-0b4d-4a83-ba52-ed177a1dfb93","Type":"ContainerStarted","Data":"64a7758aee6e464c48bda02d9d471027c2fe247fc2e8dd14f00a6e8552bb8148"} Oct 29 22:24:58.554887 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:24:58.554858 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01b1e90_e2d0_4547_a491_f91d584b6253.slice/crio-eebae58f7b7ccf3acb6c135d5e6e854b6c90c5d17de939871644214712d3184f WatchSource:0}: Error finding container eebae58f7b7ccf3acb6c135d5e6e854b6c90c5d17de939871644214712d3184f: Status 404 returned error can't find the container with id eebae58f7b7ccf3acb6c135d5e6e854b6c90c5d17de939871644214712d3184f Oct 29 22:24:58.558074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.558031 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:58.558194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.558149 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:58.558247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.558232 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:24:58.559637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.559606 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5"] Oct 29 22:24:58.567237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:58.567178 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podStartSLOduration=1.567160243 podStartE2EDuration="1.567160243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:58.562197754 +0000 UTC m=+2564.036786173" watchObservedRunningTime="2026-10-29 22:24:58.567160243 +0000 UTC m=+2564.041748662" Oct 29 22:24:59.248931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:59.241289 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:59.468929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:59.464335 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:59.501997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:59.498547 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:24:59.501997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:59.498714 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:24:59.501997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:59.498759 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 22:24:59.685930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:59.682943 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:59.708927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:59.701162 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" event={"ID":"b01b1e90-e2d0-4547-a491-f91d584b6253","Type":"ContainerStarted","Data":"d30cd9a97a780dc9e3060de21d27474e175382631905d2aa5308e51a967d4960"} Oct 29 22:24:59.708927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:59.701259 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" event={"ID":"b01b1e90-e2d0-4547-a491-f91d584b6253","Type":"ContainerStarted","Data":"eebae58f7b7ccf3acb6c135d5e6e854b6c90c5d17de939871644214712d3184f"} Oct 29 22:24:59.813981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:59.813497 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podStartSLOduration=2.8134756039999997 podStartE2EDuration="2.813475604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:59.812925314 +0000 UTC m=+2565.287513716" watchObservedRunningTime="2026-10-29 22:24:59.813475604 +0000 UTC m=+2565.288064026" Oct 29 22:24:59.885928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:24:59.885647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:00.199440 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:00.199392 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:00.202774 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:00.202745 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:25:00.204668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:00.204635 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:00.204791 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:00.204693 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:00.386589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:00.386532 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:00.497381 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:00.497296 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:25:00.497381 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:00.497354 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 22:25:00.594929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:00.589404 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:01.199956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:01.199918 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:01.200255 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:01.200221 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:01.305239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:01.305029 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:01.487991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:01.482769 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:01.514932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:01.499324 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:25:01.514932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:01.499392 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 22:25:02.140925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.140862 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:02.207432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.207386 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:02.207612 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.207459 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:02.348933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.335155 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:02.371927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.356079 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:02.371927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.359502 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:02.391059 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.384522 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:02.391059 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.384725 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 22:25:02.510095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.509509 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:25:02.510095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.509577 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 22:25:02.562934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.561563 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:02.668988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.667061 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:02.668988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.667992 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:02.849085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.849032 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:02.882278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:02.882204 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.051173 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.051061 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:03.079126 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.079080 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.196467 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.196422 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:03.196754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.196723 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:03.255809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.255745 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.492292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.473176 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 22:25:03.492292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.473566 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 22:25:03.492292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.476970 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.492292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.477339 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:25:03.495868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.494127 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.495868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.494415 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:03.540088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.527873 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:25:03.540088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.527948 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 22:25:03.540088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.528179 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.540088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.528622 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 22:25:03.618514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.618463 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.686662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.685979 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.687393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.687023 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.689947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.689921 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:03.697569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.696059 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 29 22:25:03.698062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.698040 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:25:03.837996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.708057 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 22:25:03.837996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.708125 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 22:25:03.837996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.708167 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:25:03.837996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.787023 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.910530 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.910473 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:03.942613 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:03.911559 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 22:25:04.235977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.198224 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:25:04.362944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.344567 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:04.493714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.493584 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:04.519950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.515872 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:04.569935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.545663 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:04.612145 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.605938 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:25:04.612424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.612393 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 22:25:04.618747 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.618721 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:25:04.619054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.619030 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:04.709032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.708983 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 22:25:04.928304 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.928174 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 22:25:04.929217 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.929194 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw"] Oct 29 22:25:05.085305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:04.947713 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:05.085725 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.034789 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:05.089356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.089305 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:05.115560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.115340 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:05.203580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.203479 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:05.204371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.204338 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:05.226943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.221029 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:25:05.226943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.221090 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 22:25:05.226943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.221125 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:25:05.359929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.350787 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckqvb\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-kube-api-access-ckqvb\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:05.359929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.350974 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:05.359929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.351233 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-routing-overlay-volume\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:05.359929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.351268 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-plugins-config-volume\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:05.399775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.399742 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:05.543043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.448951 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:05.604101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.463343 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:05.944567 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.928142 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-routing-overlay-volume\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:05.944874 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.944851 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-plugins-config-volume\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:05.970161 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.970116 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqvb\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-kube-api-access-ckqvb\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:05.970657 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.636945 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 29 22:25:05.970816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.636855 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 29 22:25:05.996927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.636999 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 29 22:25:05.998230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.637081 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 29 22:25:05.998422 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.637168 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-f7msq\"" Oct 29 22:25:05.998551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.637211 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ff15ac93a07f\"" Oct 29 22:25:05.998664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.732878 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:25:06.024133 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:05.463476 4952 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ff15ac93a07f: object "models-as-a-service"/"agc-cred-ff15ac93a07f" not registered Oct 29 22:25:06.024133 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:06.021085 4952 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw: object "models-as-a-service"/"agc-cred-ff15ac93a07f" not registered Oct 29 22:25:06.024133 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:06.021184 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials podName:3cca6a96-5da1-43ce-bfcb-80dcaa539e07 nodeName:}" failed. No retries permitted until 2026-10-29 22:25:06.521154865 +0000 UTC m=+2571.995743279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials") pod "payload-processing-external-model-695ccd988f-rhnqw" (UID: "3cca6a96-5da1-43ce-bfcb-80dcaa539e07") : object "models-as-a-service"/"agc-cred-ff15ac93a07f" not registered Oct 29 22:25:06.024133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.956586 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 22:25:06.024133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.955825 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-routing-overlay-volume\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:06.024133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.488436 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw"] Oct 29 22:25:06.024133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.500279 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:25:06.024133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.023527 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 22:25:06.024133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:05.956536 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-plugins-config-volume\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:06.040628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.040587 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:06.139223 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.115283 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:06.140486 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.115424 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:06.174983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.137955 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 29 22:25:06.208269 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.206526 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 29 22:25:06.208269 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.207920 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:06.208269 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.207968 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:06.213663 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.213574 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqvb\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-kube-api-access-ckqvb\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:06.359926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.356618 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 22:25:06.401180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.401133 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:06.500220 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.500125 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:25:06.513800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.513743 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 22:25:06.568966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.559293 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:06.597988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.591796 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials\") pod \"payload-processing-external-model-695ccd988f-rhnqw\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:06.665317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.657256 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:06.709961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.699780 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:06.709961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.701208 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 22:25:06.817798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.817710 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:25:06.877931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:06.868327 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:07.196715 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:07.196679 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:07.196945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:07.196747 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:07.300165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:07.300129 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw"] Oct 29 22:25:07.300631 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:25:07.300600 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cca6a96_5da1_43ce_bfcb_80dcaa539e07.slice/crio-a41140a14e57b7763c844e9d5544bfa3c9f02e1b4633adcafec70810cfff27f1 WatchSource:0}: Error finding container a41140a14e57b7763c844e9d5544bfa3c9f02e1b4633adcafec70810cfff27f1: Status 404 returned error can't find the container with id a41140a14e57b7763c844e9d5544bfa3c9f02e1b4633adcafec70810cfff27f1 Oct 29 22:25:07.506758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:07.506335 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 22:25:07.506758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:07.506399 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 22:25:07.891327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:07.891289 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:08.197032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:08.196946 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:08.197032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:08.197008 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:08.275925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:08.275866 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" event={"ID":"3cca6a96-5da1-43ce-bfcb-80dcaa539e07","Type":"ContainerStarted","Data":"a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352"} Oct 29 22:25:08.276135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:08.275953 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" event={"ID":"3cca6a96-5da1-43ce-bfcb-80dcaa539e07","Type":"ContainerStarted","Data":"a41140a14e57b7763c844e9d5544bfa3c9f02e1b4633adcafec70810cfff27f1"} Oct 29 22:25:08.276331 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:08.276272 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:08.287794 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:08.287763 4952 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ff15ac93a07f: secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:08.288055 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:08.288037 4952 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw: secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:08.288229 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:08.288216 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials podName:3cca6a96-5da1-43ce-bfcb-80dcaa539e07 nodeName:}" failed. No retries permitted until 2026-10-29 22:25:08.788195381 +0000 UTC m=+2574.262783783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials") pod "payload-processing-external-model-695ccd988f-rhnqw" (UID: "3cca6a96-5da1-43ce-bfcb-80dcaa539e07") : secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:08.343929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:08.341468 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" podStartSLOduration=5.341449187 podStartE2EDuration="5.341449187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:08.335395323 +0000 UTC m=+2573.809983743" watchObservedRunningTime="2026-10-29 22:25:08.341449187 +0000 UTC m=+2573.816037605" Oct 29 22:25:08.537338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:08.534260 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:25:08.537338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:08.536002 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:25:08.550478 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:08.550447 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:25:08.897931 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:08.857753 4952 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ff15ac93a07f: secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:08.944032 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:08.917741 4952 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw: secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:08.944032 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:08.917827 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials podName:3cca6a96-5da1-43ce-bfcb-80dcaa539e07 nodeName:}" failed. No retries permitted until 2026-10-29 22:25:09.917806794 +0000 UTC m=+2575.392395209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials") pod "payload-processing-external-model-695ccd988f-rhnqw" (UID: "3cca6a96-5da1-43ce-bfcb-80dcaa539e07") : secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:09.078614 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:09.078567 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:09.197100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:09.197003 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:09.197385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:09.197089 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:09.251953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:09.251874 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:09.428529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:09.428490 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:09.611939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:09.610930 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:09.899137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:09.899020 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:10.007732 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:10.007700 4952 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ff15ac93a07f: secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:10.007732 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:10.007728 4952 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw: secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:10.008033 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:10.007786 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials podName:3cca6a96-5da1-43ce-bfcb-80dcaa539e07 nodeName:}" failed. No retries permitted until 2026-10-29 22:25:12.007770421 +0000 UTC m=+2577.482358821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials") pod "payload-processing-external-model-695ccd988f-rhnqw" (UID: "3cca6a96-5da1-43ce-bfcb-80dcaa539e07") : secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:10.196566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:10.196473 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:10.196566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:10.196546 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:10.569518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:10.569431 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:10.786943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:10.785513 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:10.974940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:10.973149 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:11.196451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:11.196413 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xfcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 29 22:25:11.196773 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:11.196722 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 29 22:25:11.680638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:11.680593 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:11.901331 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:11.901287 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:12.102836 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:12.102794 4952 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ff15ac93a07f: secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:12.102836 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:12.102833 4952 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw: secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:12.103988 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:12.102918 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials podName:3cca6a96-5da1-43ce-bfcb-80dcaa539e07 nodeName:}" failed. No retries permitted until 2026-10-29 22:25:16.102882417 +0000 UTC m=+2581.577470836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials") pod "payload-processing-external-model-695ccd988f-rhnqw" (UID: "3cca6a96-5da1-43ce-bfcb-80dcaa539e07") : secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:12.341683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.341645 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:12.541560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.541466 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:12.894327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.894291 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc"] Oct 29 22:25:12.900129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.900085 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:12.971402 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.971346 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:12.972715 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.972687 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2dr\" (UniqueName: \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-kube-api-access-5t2dr\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:12.972927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.972889 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:12.973206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.973185 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:12.973441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.973421 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:12.973723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.973651 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:12.973931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.973891 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:12.974138 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.974118 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:12.974308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.974290 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:12.983754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:12.983518 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc"] Oct 29 22:25:13.075834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.075795 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076019 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.075965 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076033 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076057 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076107 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076147 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076283 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076299 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076369 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2dr\" (UniqueName: \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-kube-api-access-5t2dr\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076428 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076582 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076767 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.076923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076880 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.077028 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.076941 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.079282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.079260 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.079559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.079538 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.085658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.085627 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:25:13.086516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.086495 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:25:13.087340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.087321 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:25:13.104113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.104055 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.104509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.104251 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2dr\" (UniqueName: \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-kube-api-access-5t2dr\") pod \"maas-default-gateway-openshift-default-6f54779779-8m9fc\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.187976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.183660 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:13.297590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.297554 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:13.815188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.815096 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw"] Oct 29 22:25:13.846369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.846337 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:13.997005 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.996860 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:25:13.997360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.997326 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc"] Oct 29 22:25:13.997453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.997324 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:25:13.997531 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.997487 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:25:13.999226 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:13.998801 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:14.393039 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:14.385706 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:14.593753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:14.593708 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" event={"ID":"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a","Type":"ContainerStarted","Data":"5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f"} Oct 29 22:25:14.594006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:14.593951 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" event={"ID":"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a","Type":"ContainerStarted","Data":"be045623ee683658e855b400af432fd1760dd07fb601f4e907c75720fb136f4c"} Oct 29 22:25:14.659896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:14.659588 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podStartSLOduration=2.659568914 podStartE2EDuration="2.659568914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:14.64784687 +0000 UTC m=+2580.122435293" watchObservedRunningTime="2026-10-29 22:25:14.659568914 +0000 UTC m=+2580.134157334" Oct 29 22:25:15.326940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:15.325281 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:15.349929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:15.331687 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8m9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 29 22:25:15.349929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:15.331749 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 29 22:25:15.349929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:15.332579 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:15.498610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:15.498568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:15.585984 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:15.584131 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:25:15.691722 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:15.691681 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:15.988151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:15.988003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:16.173929 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:16.173203 4952 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ff15ac93a07f: secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:16.173929 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:16.173240 4952 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw: secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:16.173929 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:16.173305 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials podName:3cca6a96-5da1-43ce-bfcb-80dcaa539e07 nodeName:}" failed. No retries permitted until 2026-10-29 22:25:24.17328372 +0000 UTC m=+2589.647872130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials") pod "payload-processing-external-model-695ccd988f-rhnqw" (UID: "3cca6a96-5da1-43ce-bfcb-80dcaa539e07") : secret "agc-cred-ff15ac93a07f" not found Oct 29 22:25:16.333931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:16.329369 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8m9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 29 22:25:16.333931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:16.329432 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 29 22:25:16.545928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:16.545352 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:16.896971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:16.894020 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:17.356939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:17.354608 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8m9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 29 22:25:17.356939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:17.354676 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 29 22:25:17.708346 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:17.708221 4952 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.066s" Oct 29 22:25:18.253994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:18.253887 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:18.324467 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:18.324392 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8m9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 29 22:25:18.326240 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:18.326204 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 29 22:25:18.823944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:18.823857 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" podUID="3cca6a96-5da1-43ce-bfcb-80dcaa539e07" containerName="payload-processing" containerID="cri-o://a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352" gracePeriod=25 Oct 29 22:25:19.366021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.345297 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8m9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 29 22:25:19.366021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.345360 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 29 22:25:19.420698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.420655 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:19.487266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.487230 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-kube-api-access-ckqvb\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-kube-api-access-ckqvb\") pod \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " Oct 29 22:25:19.487503 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.487337 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-routing-overlay-volume\") pod \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " Oct 29 22:25:19.487503 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.487416 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-plugins-config-volume\") pod \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " Oct 29 22:25:19.489812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.487724 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials\") pod \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\" (UID: \"3cca6a96-5da1-43ce-bfcb-80dcaa539e07\") " Oct 29 22:25:19.489812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.488029 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-routing-overlay-volume" pod "3cca6a96-5da1-43ce-bfcb-80dcaa539e07" (UID: "3cca6a96-5da1-43ce-bfcb-80dcaa539e07"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:19.489812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.488390 4952 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-routing-overlay-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.489812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.488710 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-plugins-config-volume" pod "3cca6a96-5da1-43ce-bfcb-80dcaa539e07" (UID: "3cca6a96-5da1-43ce-bfcb-80dcaa539e07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:19.490333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.490306 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-kube-api-access-ckqvb" pod "3cca6a96-5da1-43ce-bfcb-80dcaa539e07" (UID: "3cca6a96-5da1-43ce-bfcb-80dcaa539e07"). InnerVolumeSpecName "kube-api-access-ckqvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:19.491106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.491064 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials" pod "3cca6a96-5da1-43ce-bfcb-80dcaa539e07" (UID: "3cca6a96-5da1-43ce-bfcb-80dcaa539e07"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:19.589702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.589662 4952 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-provider-credentials\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.589702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.589695 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckqvb\" (UniqueName: \"kubernetes.io/projected/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-kube-api-access-ckqvb\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.589702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.589705 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cca6a96-5da1-43ce-bfcb-80dcaa539e07-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:19.810947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.806327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:19.979059 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.978508 4952 generic.go:356] "Generic (PLEG): container finished" podID="3cca6a96-5da1-43ce-bfcb-80dcaa539e07" containerID="a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352" exitCode=0 Oct 29 22:25:19.979059 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.978843 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" event={"ID":"3cca6a96-5da1-43ce-bfcb-80dcaa539e07","Type":"ContainerDied","Data":"a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352"} Oct 29 22:25:19.979059 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.979059 4952 scope.go:122] "RemoveContainer" containerID="a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352" Oct 29 22:25:19.979683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.979241 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" Oct 29 22:25:19.980295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.978899 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw" event={"ID":"3cca6a96-5da1-43ce-bfcb-80dcaa539e07","Type":"ContainerDied","Data":"a41140a14e57b7763c844e9d5544bfa3c9f02e1b4633adcafec70810cfff27f1"} Oct 29 22:25:19.999216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.999192 4952 scope.go:122] "RemoveContainer" containerID="a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352" Oct 29 22:25:19.999561 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:19.999541 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352\": container with ID starting with a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352 not found: ID does not exist" containerID="a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352" Oct 29 22:25:19.999670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:19.999571 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352"} err="failed to get container status \"a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352\": rpc error: code = NotFound desc = could not find container \"a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352\": container with ID starting with a78e285a06e7a6f01f0f586113e6cdf948ace5bbc4a51caece7748acd55c9352 not found: ID does not exist" Oct 29 22:25:20.006954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:20.005415 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:20.139928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:20.126971 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw"] Oct 29 22:25:20.280977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:20.274860 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-695ccd988f-rhnqw"] Oct 29 22:25:20.391982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:20.383036 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8m9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 29 22:25:20.391982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:20.383119 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 29 22:25:20.661971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:20.658634 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cca6a96-5da1-43ce-bfcb-80dcaa539e07" path="/var/lib/kubelet/pods/3cca6a96-5da1-43ce-bfcb-80dcaa539e07/volumes" Oct 29 22:25:20.990003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:20.988423 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:21.331928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:21.326294 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8m9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 29 22:25:21.331928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:21.326355 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 29 22:25:22.337704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:22.337662 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8m9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 29 22:25:22.338835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:22.338795 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 29 22:25:23.166928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.162072 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:23.322984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.322948 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8m9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 29 22:25:23.323163 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.323003 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 29 22:25:23.719249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.719216 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j"] Oct 29 22:25:23.720046 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.720025 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3cca6a96-5da1-43ce-bfcb-80dcaa539e07" containerName="payload-processing" Oct 29 22:25:23.720046 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.720046 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cca6a96-5da1-43ce-bfcb-80dcaa539e07" containerName="payload-processing" Oct 29 22:25:23.720214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.720191 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cca6a96-5da1-43ce-bfcb-80dcaa539e07" containerName="payload-processing" Oct 29 22:25:23.728507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.728470 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.746538 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.746330 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j"] Oct 29 22:25:23.753361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.751214 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-529e60\"/\"openshift-service-ca.crt\"" Oct 29 22:25:23.753361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.751239 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-529e60\"/\"kube-root-ca.crt\"" Oct 29 22:25:23.772949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.772029 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-529e60\"/\"explicit-model-529e60-kserve-self-signed-certs\"" Oct 29 22:25:23.772949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.772200 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-529e60\"/\"default-dockercfg-gqt6p\"" Oct 29 22:25:23.861701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.861663 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7pw\" (UniqueName: \"kubernetes.io/projected/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-kube-api-access-nw7pw\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.862202 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.862174 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-home\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.862503 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.862440 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-dshm\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.862651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.862635 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tmp-dir\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.862852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.862834 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tls-certs\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.863144 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.863126 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-model-cache\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.866236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.866070 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:23.984197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.982048 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7pw\" (UniqueName: \"kubernetes.io/projected/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-kube-api-access-nw7pw\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.984197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.982242 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-home\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.984197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.982309 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-dshm\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.984197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.982338 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tmp-dir\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.984197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.982404 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tls-certs\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.984197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.982517 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-model-cache\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.984197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.983164 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-model-cache\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.984197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.983756 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-home\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:23.986100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:23.986073 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tmp-dir\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:24.183673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:24.183599 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7pw\" (UniqueName: \"kubernetes.io/projected/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-kube-api-access-nw7pw\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:24.190438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:24.190410 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-dshm\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:24.195943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:24.194238 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tls-certs\") pod \"explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:24.323029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:24.322505 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8m9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 29 22:25:24.323029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:24.322570 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 29 22:25:24.357602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:24.357566 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:24.705285 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:25:24.705253 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e6cd2d4_da8b_48af_b736_9b9cff3c3462.slice/crio-3af8bd9da0ed7d5cc5074d33bfd88c4038d8cd0f7b5d0b947d6147ab484ccbdf WatchSource:0}: Error finding container 3af8bd9da0ed7d5cc5074d33bfd88c4038d8cd0f7b5d0b947d6147ab484ccbdf: Status 404 returned error can't find the container with id 3af8bd9da0ed7d5cc5074d33bfd88c4038d8cd0f7b5d0b947d6147ab484ccbdf Oct 29 22:25:24.716989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:24.716959 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j"] Oct 29 22:25:25.166927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:25.166772 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:25.181930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:25.180532 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" event={"ID":"9e6cd2d4-da8b-48af-b736-9b9cff3c3462","Type":"ContainerStarted","Data":"1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd"} Oct 29 22:25:25.181930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:25.180596 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" event={"ID":"9e6cd2d4-da8b-48af-b736-9b9cff3c3462","Type":"ContainerStarted","Data":"3af8bd9da0ed7d5cc5074d33bfd88c4038d8cd0f7b5d0b947d6147ab484ccbdf"} Oct 29 22:25:25.248961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:25.247010 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" podStartSLOduration=2.246990669 podStartE2EDuration="2.246990669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:25.232342285 +0000 UTC m=+2590.706930709" watchObservedRunningTime="2026-10-29 22:25:25.246990669 +0000 UTC m=+2590.721579089" Oct 29 22:25:25.367925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:25.359774 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:26.136536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:26.136500 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:26.137203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:26.137177 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:26.138500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:26.138478 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:25:27.014637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:27.014588 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:28.773933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:28.766739 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:30.488928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:30.483603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:32.345009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:32.342046 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:34.037928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:34.037043 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:34.358397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:34.358352 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:34.372296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:34.372269 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:35.291480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.291447 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:35.545952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.545822 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv"] Oct 29 22:25:35.551829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.551802 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.577204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.577174 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv"] Oct 29 22:25:35.717404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.717367 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.717570 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.717537 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.717617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.717573 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44rrg\" (UniqueName: \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-kube-api-access-44rrg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.717659 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.717631 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.717692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.717667 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.717825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.717805 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82c80abc-aa30-4ae0-b724-6dff35e30669-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.717879 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.717867 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.818831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.818746 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82c80abc-aa30-4ae0-b724-6dff35e30669-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.819067 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.818862 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.819067 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.819026 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.819304 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.819282 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.819409 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.819335 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44rrg\" (UniqueName: \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-kube-api-access-44rrg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.819477 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.819443 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.819593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.819568 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.820560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.820534 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.822166 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.822138 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82c80abc-aa30-4ae0-b724-6dff35e30669-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.822487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.822466 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.822610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.822565 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.822842 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.822824 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.845639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.845608 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44rrg\" (UniqueName: \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-kube-api-access-44rrg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.846151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.846128 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rlhrv\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.875974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.873142 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:35.891093 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:35.891055 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:36.238593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:36.238558 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv"] Oct 29 22:25:36.239053 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:25:36.239024 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c80abc_aa30_4ae0_b724_6dff35e30669.slice/crio-f54ee598997779469e562ebdd08e90553b981ca1914e4d766bb686c2aa9800e1 WatchSource:0}: Error finding container f54ee598997779469e562ebdd08e90553b981ca1914e4d766bb686c2aa9800e1: Status 404 returned error can't find the container with id f54ee598997779469e562ebdd08e90553b981ca1914e4d766bb686c2aa9800e1 Oct 29 22:25:36.242513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:36.242475 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:25:36.242618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:36.242553 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:25:36.292393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:36.292355 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" event={"ID":"82c80abc-aa30-4ae0-b724-6dff35e30669","Type":"ContainerStarted","Data":"f54ee598997779469e562ebdd08e90553b981ca1914e4d766bb686c2aa9800e1"} Oct 29 22:25:37.305599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:37.305556 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" event={"ID":"82c80abc-aa30-4ae0-b724-6dff35e30669","Type":"ContainerStarted","Data":"c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211"} Oct 29 22:25:37.306849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:37.306826 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:37.310820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:37.310786 4952 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rlhrv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:25:37.311023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:37.310880 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" podUID="82c80abc-aa30-4ae0-b724-6dff35e30669" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:25:37.406566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:37.406490 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" podStartSLOduration=2.406466454 podStartE2EDuration="2.406466454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:37.369476917 +0000 UTC m=+2602.844065351" watchObservedRunningTime="2026-10-29 22:25:37.406466454 +0000 UTC m=+2602.881054875" Oct 29 22:25:37.748926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:37.746539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:38.318070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:38.318037 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:25:39.593924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:39.590572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:39.734052 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:39.734007 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:25:39.777676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:39.777631 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:39.963300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:39.963199 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:42.699130 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:42.699074 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 29 22:25:42.722819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:42.722772 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 29 22:25:42.818835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:42.818760 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:42.913939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:42.913210 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j"] Oct 29 22:25:43.083980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:43.077703 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 29 22:25:43.083980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:43.078137 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 29 22:25:43.083980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:43.078207 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:43.097936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:43.092221 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:43.131574 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:43.131504 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:25:43.144794 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:43.144708 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:43.208134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:43.208023 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 29 22:25:43.263963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:43.261612 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:43.340532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:43.339966 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" podUID="9e6cd2d4-da8b-48af-b736-9b9cff3c3462" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:25:43.438619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:43.438583 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:45.585414 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:45.585314 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:25:45.908414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:45.907416 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:47.884927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:47.884434 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:49.492801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:49.492764 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:51.178639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:51.178602 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:52.759033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:52.758990 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:53.175070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:53.175019 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:53.685945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:53.685892 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s"] Oct 29 22:25:53.917225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:53.917176 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:54.090326 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:54.090155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:54.603928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:54.574939 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:56.748932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:56.747350 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:58.209406 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.209340 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" podUID="9e6cd2d4-da8b-48af-b736-9b9cff3c3462" containerName="main" containerID="cri-o://1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd" gracePeriod=45 Oct 29 22:25:58.280853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.280804 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" podUID="9e6cd2d4-da8b-48af-b736-9b9cff3c3462" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": dial tcp 10.132.0.177:8000: connect: connection refused" Oct 29 22:25:58.434471 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.434306 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:58.608810 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.608784 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:58.687079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.686998 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" podUID="99bc6aed-e198-4f9f-8a2b-54eaaea369ec" containerName="payload-processing" containerID="cri-o://7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365" gracePeriod=25 Oct 29 22:25:58.721157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.721128 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tls-certs\") pod \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " Oct 29 22:25:58.721286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.721186 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-kube-api-access-nw7pw\" (UniqueName: \"kubernetes.io/projected/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-kube-api-access-nw7pw\") pod \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " Oct 29 22:25:58.721286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.721268 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tmp-dir\") pod \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " Oct 29 22:25:58.721498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.721479 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-home\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-home\") pod \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " Oct 29 22:25:58.721590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.721557 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tmp-dir" pod "9e6cd2d4-da8b-48af-b736-9b9cff3c3462" (UID: "9e6cd2d4-da8b-48af-b736-9b9cff3c3462"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:58.721590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.721586 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-dshm\") pod \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " Oct 29 22:25:58.721705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.721694 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-model-cache\") pod \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\" (UID: \"9e6cd2d4-da8b-48af-b736-9b9cff3c3462\") " Oct 29 22:25:58.721772 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.721737 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-home" pod "9e6cd2d4-da8b-48af-b736-9b9cff3c3462" (UID: "9e6cd2d4-da8b-48af-b736-9b9cff3c3462"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:58.722742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.722398 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-model-cache" pod "9e6cd2d4-da8b-48af-b736-9b9cff3c3462" (UID: "9e6cd2d4-da8b-48af-b736-9b9cff3c3462"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:58.722742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.722674 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:58.722742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.722697 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:58.722742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.722710 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:58.724195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.724166 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-kube-api-access-nw7pw" pod "9e6cd2d4-da8b-48af-b736-9b9cff3c3462" (UID: "9e6cd2d4-da8b-48af-b736-9b9cff3c3462"). InnerVolumeSpecName "kube-api-access-nw7pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:58.724325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.724241 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tls-certs" pod "9e6cd2d4-da8b-48af-b736-9b9cff3c3462" (UID: "9e6cd2d4-da8b-48af-b736-9b9cff3c3462"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:25:58.724457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.724433 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-dshm" pod "9e6cd2d4-da8b-48af-b736-9b9cff3c3462" (UID: "9e6cd2d4-da8b-48af-b736-9b9cff3c3462"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:25:58.825327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.825295 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:58.825327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.825332 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:58.825577 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.825347 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7pw\" (UniqueName: \"kubernetes.io/projected/9e6cd2d4-da8b-48af-b736-9b9cff3c3462-kube-api-access-nw7pw\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:58.982578 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:58.982551 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:25:59.099124 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.099037 4952 generic.go:356] "Generic (PLEG): container finished" podID="99bc6aed-e198-4f9f-8a2b-54eaaea369ec" containerID="7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365" exitCode=0 Oct 29 22:25:59.099394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.099366 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" event={"ID":"99bc6aed-e198-4f9f-8a2b-54eaaea369ec","Type":"ContainerDied","Data":"7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365"} Oct 29 22:25:59.099480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.099433 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" event={"ID":"99bc6aed-e198-4f9f-8a2b-54eaaea369ec","Type":"ContainerDied","Data":"40555f651f3666dfa8ef7940cd8c959ef3514dfd66c1bf6c65e4dbf4d5f42310"} Oct 29 22:25:59.099480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.099463 4952 scope.go:122] "RemoveContainer" containerID="7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365" Oct 29 22:25:59.099637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.099620 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s" Oct 29 22:25:59.103104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.102835 4952 generic.go:356] "Generic (PLEG): container finished" podID="9e6cd2d4-da8b-48af-b736-9b9cff3c3462" containerID="1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd" exitCode=0 Oct 29 22:25:59.103104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.102884 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" event={"ID":"9e6cd2d4-da8b-48af-b736-9b9cff3c3462","Type":"ContainerDied","Data":"1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd"} Oct 29 22:25:59.103104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.102964 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" event={"ID":"9e6cd2d4-da8b-48af-b736-9b9cff3c3462","Type":"ContainerDied","Data":"3af8bd9da0ed7d5cc5074d33bfd88c4038d8cd0f7b5d0b947d6147ab484ccbdf"} Oct 29 22:25:59.103104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.103026 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j" Oct 29 22:25:59.117117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.117095 4952 scope.go:122] "RemoveContainer" containerID="7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365" Oct 29 22:25:59.118247 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:59.118207 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365\": container with ID starting with 7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365 not found: ID does not exist" containerID="7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365" Oct 29 22:25:59.118350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.118250 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365"} err="failed to get container status \"7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365\": rpc error: code = NotFound desc = could not find container \"7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365\": container with ID starting with 7d65df2b98b36b2422486b5b7805e01981ce2e4a88628a507d0ce4fda2109365 not found: ID does not exist" Oct 29 22:25:59.118350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.118275 4952 scope.go:122] "RemoveContainer" containerID="1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd" Oct 29 22:25:59.130985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.130963 4952 scope.go:122] "RemoveContainer" containerID="1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd" Oct 29 22:25:59.131302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.131127 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-plugins-config-volume\") pod \"99bc6aed-e198-4f9f-8a2b-54eaaea369ec\" (UID: \"99bc6aed-e198-4f9f-8a2b-54eaaea369ec\") " Oct 29 22:25:59.131302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.131163 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-kube-api-access-8vbfv\" (UniqueName: \"kubernetes.io/projected/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-kube-api-access-8vbfv\") pod \"99bc6aed-e198-4f9f-8a2b-54eaaea369ec\" (UID: \"99bc6aed-e198-4f9f-8a2b-54eaaea369ec\") " Oct 29 22:25:59.131556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.131536 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-plugins-config-volume" pod "99bc6aed-e198-4f9f-8a2b-54eaaea369ec" (UID: "99bc6aed-e198-4f9f-8a2b-54eaaea369ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:25:59.131666 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:25:59.131629 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd\": container with ID starting with 1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd not found: ID does not exist" containerID="1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd" Oct 29 22:25:59.131742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.131672 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd"} err="failed to get container status \"1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd\": rpc error: code = NotFound desc = could not find container \"1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd\": container with ID starting with 1484f25fe8c2d43f818b216bbfa3789fff12c1f5ac7bc6cf0a9d8fd6bc2951bd not found: ID does not exist" Oct 29 22:25:59.132151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.132131 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.134610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.134582 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-kube-api-access-8vbfv" pod "99bc6aed-e198-4f9f-8a2b-54eaaea369ec" (UID: "99bc6aed-e198-4f9f-8a2b-54eaaea369ec"). InnerVolumeSpecName "kube-api-access-8vbfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:25:59.162263 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.162217 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j"] Oct 29 22:25:59.170205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.170170 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-529e60/explicit-model-529e60-kserve-6c6bdfbcc7-z5z4j"] Oct 29 22:25:59.234859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.234825 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vbfv\" (UniqueName: \"kubernetes.io/projected/99bc6aed-e198-4f9f-8a2b-54eaaea369ec-kube-api-access-8vbfv\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:25:59.461719 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.461677 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s"] Oct 29 22:25:59.465545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:25:59.465264 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-529e60-7f75d56968-9lk8s"] Oct 29 22:26:00.094389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:00.094348 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:00.646805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:00.646773 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99bc6aed-e198-4f9f-8a2b-54eaaea369ec" path="/var/lib/kubelet/pods/99bc6aed-e198-4f9f-8a2b-54eaaea369ec/volumes" Oct 29 22:26:00.647378 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:00.647361 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6cd2d4-da8b-48af-b736-9b9cff3c3462" path="/var/lib/kubelet/pods/9e6cd2d4-da8b-48af-b736-9b9cff3c3462/volumes" Oct 29 22:26:01.976955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:01.971082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:03.627925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:03.621991 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:05.450979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:05.442890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:07.446830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:07.446786 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:09.249288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:09.249243 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:10.872927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:10.872424 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:11.051808 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:11.051761 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:11.330197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:11.330158 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:12.525429 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:12.525395 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:14.076781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:14.076722 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:15.073773 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:15.073728 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:15.278160 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:15.277556 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:15.532471 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:26:15.532357 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:26:15.828225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:15.828166 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:17.161846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:17.161812 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:19.073058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:19.072833 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:20.733264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:20.733229 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:22.571164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:22.571125 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:23.216332 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:23.216299 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:23.390548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:23.390509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:23.645823 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:23.645775 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:24.599506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:24.599462 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:25.674630 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:25.674588 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:26.241843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:26.241808 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:26.490661 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:26.490624 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:28.549928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:28.545090 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:29.761954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:29.761916 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:29.959465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:29.959430 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:30.790824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:30.789875 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:31.650640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:31.650603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:31.829890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:31.829829 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:32.882543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:32.882507 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:34.768139 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:34.768099 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:37.088930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:37.088286 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:26:37.111506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:37.111459 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:26:37.116121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:37.116058 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:26:37.147930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:37.147139 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 22:26:37.214315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:37.214263 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 29 22:26:37.336932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:37.331721 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:26:37.360139 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:37.360026 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:26:37.832555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:37.832497 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:39.668790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:39.668751 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:41.287013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:41.283558 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:43.183500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:43.183460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:44.970566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:44.970527 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:45.529148 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:26:45.529107 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:26:46.599512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:46.599447 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:46.941224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:46.941118 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:48.194633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:48.194597 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:48.551987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:48.464352 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:49.383307 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:49.383266 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:49.565930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:49.563476 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:51.322683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:51.322647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:51.580502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:51.580468 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:52.145934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:52.138177 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:54.238476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:54.238440 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:55.960822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:55.960770 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:57.759312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:57.759219 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:58.162608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:58.162564 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:59.580600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:26:59.580560 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:01.394954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:01.391656 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:03.079427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:03.079388 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:03.390822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:03.390722 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:03.627505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:03.627468 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:05.477928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:05.473387 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:07.202925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:07.202811 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:08.517511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:08.517475 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:27:09.018081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:09.016439 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:10.653614 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:10.653568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:11.016561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:11.016436 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:11.443849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:11.443799 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:12.239738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:12.239702 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:13.840874 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:13.840837 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:15.210864 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:15.210821 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:15.437887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:15.437851 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:15.892310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:15.892274 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:15.989937 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:27:15.987831 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:27:16.022924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:16.021216 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:27:16.041608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:16.041583 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:27:16.221705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:16.221610 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:17.893934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:17.890821 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:19.673977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:19.669405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:21.550599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:21.550562 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:22.882384 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:22.882344 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:23.065313 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:23.065277 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:23.246596 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:23.246500 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:23.436956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:23.434595 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:25.258106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:25.258067 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:27.030940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:27.029707 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:29.057324 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:29.054503 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:30.733832 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:30.733795 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:30.917864 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:30.917830 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:31.131992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:31.131955 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:31.313975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:31.313935 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:31.499269 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:31.499175 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:31.685416 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:31.685377 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:31.934964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:31.934929 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:32.134938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:32.132085 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:32.773636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:32.773586 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:32.941386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:32.941353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:33.191920 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:33.191870 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:34.149830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:34.149788 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:34.404588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:34.404504 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:35.852926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:35.846590 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:37.540511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:37.540473 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:39.377038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:39.377000 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:40.956306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:40.956229 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:42.675108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:42.666842 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:44.804489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:44.804447 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:45.530519 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:27:45.530485 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:27:46.454556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:46.454515 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:48.323858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:48.323821 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:48.644625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:48.644539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:48.828884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:48.828841 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:51.294937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:51.293447 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:51.551375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:51.551278 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:53.332856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:53.332816 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:54.346634 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:54.346546 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:54.563107 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:54.563071 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:55.523551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:55.523509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:56.240975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:56.238470 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:57.081562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:57.081517 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:57.267563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:57.267483 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:58.182899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:58.182859 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:59.950946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:27:59.948604 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:00.316967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:00.316824 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:00.502359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:00.502323 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:00.961825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:00.960698 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:02.053520 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:02.053475 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:03.600681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:03.600645 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:03.967550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:03.967441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:04.139038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:04.138959 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:05.256262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:05.256217 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:06.807383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:06.807347 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:08.299589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:08.299543 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:08.474959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:08.474920 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:08.643767 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:08.643724 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:10.052517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:10.052475 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:11.137641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:11.137603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:11.563558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:11.563460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:11.759411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:11.759371 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:12.823523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:12.823482 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:12.997999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:12.997948 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:13.313865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:13.313828 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:14.993760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:14.993718 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:15.166463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:15.166427 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:15.328878 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:15.328840 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:15.502345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:15.502309 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:15.529804 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:28:15.529768 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:28:15.776983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:15.776478 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:16.397626 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:16.397585 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:16.602936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:16.580593 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:17.633739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:17.633683 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:17.634721 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:17.634679 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 29 22:28:17.741930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:17.718997 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:17.741930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:17.719437 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:17.822929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:17.820038 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:17.907987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:17.893830 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:17.975392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:17.968890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:18.168031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:18.167895 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:18.354819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:18.354775 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:18.953926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:18.949708 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:19.113144 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:19.113108 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:19.533990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:19.533951 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:19.711131 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:19.711090 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:20.164860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:20.164825 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:20.828863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:20.828826 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:20.998250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:20.998212 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:22.509009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:22.508969 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:24.068944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:24.062848 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:25.631950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:25.631893 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:26.058675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:26.058587 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:26.236357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:26.236319 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:27.212283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:27.212243 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:27.385216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:27.385177 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:27.770469 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:27.770435 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:28.274185 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:28.274146 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:28.462512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:28.462471 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:29.295581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:29.273275 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:31.001740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:31.001700 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:31.652509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:31.652466 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:31.820660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:31.820626 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:32.859927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:32.853724 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:33.423967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:33.423879 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:33.548342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:33.548302 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:28:33.615916 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:33.615865 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:34.028927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:34.024824 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:34.199777 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:34.199734 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:34.558314 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:34.558278 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:34.772930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:34.771842 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:35.142931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:35.140467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:35.564224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:35.564117 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:35.824116 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:35.823047 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:35.887253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:35.886049 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:36.098067 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:36.096278 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:36.152932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:36.128058 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:28:36.209927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:36.208140 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:28:37.764866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:37.764681 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:38.017118 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:38.017036 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:39.435894 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:39.435851 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:39.665364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:39.664669 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:39.837534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:39.837497 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:40.074891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:40.074841 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:40.353928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:40.330461 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:41.532148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:41.532106 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:41.704349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:41.704316 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:42.294230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:42.286846 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:43.945930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:43.941001 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:44.157385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:44.157345 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:44.386318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:44.386283 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:45.530547 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:28:45.530511 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:28:45.820550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:45.820453 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:46.003877 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:46.003839 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:47.702864 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:47.702779 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:49.486652 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:49.486613 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:51.236368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:51.236329 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:52.940949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:52.936992 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:54.671482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:54.671443 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:56.389930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:56.386583 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:57.022376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:57.022331 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:57.196988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:57.196954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:58.131920 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:58.131865 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:59.876021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:28:59.875983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:01.661552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:01.661512 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:03.315371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:03.315332 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:05.075929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:05.060008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:07.018193 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:07.018151 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:08.936154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:08.936111 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:10.631930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:10.631725 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:11.264081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:11.264045 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:12.437929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:12.433082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:14.117064 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:14.117026 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:15.529204 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:29:15.529167 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:29:15.790710 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:15.790623 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:17.522592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:17.521579 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:17.873727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:17.873687 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:18.048202 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:18.048165 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:19.225122 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:19.225084 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:20.051710 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:20.051664 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:20.232930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:20.232609 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:20.429592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:20.429558 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:20.824927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:20.824879 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:20.997062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:20.996393 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:21.176323 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:21.176202 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:21.345864 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:21.345820 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:21.587194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:21.587158 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:21.837144 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:21.837109 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:22.306260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:22.306223 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:22.505705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:22.505669 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:23.918555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:23.918479 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:24.223542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:24.223447 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:24.970699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:24.970661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:25.175403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:25.175362 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:25.361700 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:25.361666 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:25.602937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:25.593851 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:26.837979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:26.836966 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:28.667935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:28.667243 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:30.369930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:30.369651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:32.120664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:32.120591 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:33.578993 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.578960 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l"] Oct 29 22:29:33.579757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.579723 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9e6cd2d4-da8b-48af-b736-9b9cff3c3462" containerName="main" Oct 29 22:29:33.579814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.579758 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6cd2d4-da8b-48af-b736-9b9cff3c3462" containerName="main" Oct 29 22:29:33.579814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.579776 4952 cpu_manager.go:414] "containerMap: removing container" podUID="99bc6aed-e198-4f9f-8a2b-54eaaea369ec" containerName="payload-processing" Oct 29 22:29:33.579814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.579782 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bc6aed-e198-4f9f-8a2b-54eaaea369ec" containerName="payload-processing" Oct 29 22:29:33.579925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.579913 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6cd2d4-da8b-48af-b736-9b9cff3c3462" containerName="main" Oct 29 22:29:33.579964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.579924 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bc6aed-e198-4f9f-8a2b-54eaaea369ec" containerName="payload-processing" Oct 29 22:29:33.583757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.583721 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.601944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.601922 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c06a2c-serving-cert\"" Oct 29 22:29:33.628681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.628642 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l"] Oct 29 22:29:33.704800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.704757 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-metrics-certs\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.705035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.705006 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hfbj\" (UniqueName: \"kubernetes.io/projected/5a693545-a2b8-41c3-984c-74905ea0b217-kube-api-access-4hfbj\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.705122 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.705106 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-maas-api-tls\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.705338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.705313 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a693545-a2b8-41c3-984c-74905ea0b217-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.791699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.791652 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:33.809768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.808045 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-metrics-certs\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.809768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.808197 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hfbj\" (UniqueName: \"kubernetes.io/projected/5a693545-a2b8-41c3-984c-74905ea0b217-kube-api-access-4hfbj\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.809768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.808270 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-maas-api-tls\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.809768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.808348 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a693545-a2b8-41c3-984c-74905ea0b217-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.809768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.809315 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a693545-a2b8-41c3-984c-74905ea0b217-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.810684 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:29:33.810663 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-serving-cert: secret "maas-api-e2e-inf-a-c06a2c-serving-cert" not found Oct 29 22:29:33.811009 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:29:33.810991 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-maas-api-tls podName:5a693545-a2b8-41c3-984c-74905ea0b217 nodeName:}" failed. No retries permitted until 2026-10-29 22:29:34.310968953 +0000 UTC m=+2839.785557357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-maas-api-tls") pod "maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" (UID: "5a693545-a2b8-41c3-984c-74905ea0b217") : secret "maas-api-e2e-inf-a-c06a2c-serving-cert" not found Oct 29 22:29:33.846036 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.845685 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-metrics-certs\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.853832 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.853799 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hfbj\" (UniqueName: \"kubernetes.io/projected/5a693545-a2b8-41c3-984c-74905ea0b217-kube-api-access-4hfbj\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:33.964688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:33.964649 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:34.147229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:34.147120 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:34.317514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:34.317479 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-maas-api-tls\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:34.320203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:34.320176 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-maas-api-tls\") pod \"maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:34.503506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:34.503416 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:35.027539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:35.027512 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:29:35.036157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:35.036123 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l"] Oct 29 22:29:35.229077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:35.229033 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" event={"ID":"5a693545-a2b8-41c3-984c-74905ea0b217","Type":"ContainerStarted","Data":"47729770a8e030ef1b1513429b0d60c8f58ccc57d229608bd457ed6c7c7c8730"} Oct 29 22:29:35.766006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:35.764420 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:36.240781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:36.240730 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" event={"ID":"5a693545-a2b8-41c3-984c-74905ea0b217","Type":"ContainerStarted","Data":"5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1"} Oct 29 22:29:36.241251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:36.240854 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:36.288043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:36.287989 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" podStartSLOduration=2.899812182 podStartE2EDuration="3.287974053s" totalImagesPullingTime="388.161871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:36.286975976 +0000 UTC m=+2841.761564391" watchObservedRunningTime="2026-10-29 22:29:36.287974053 +0000 UTC m=+2841.762562473" Oct 29 22:29:37.552543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:37.547487 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:38.358042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:38.358008 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:29:39.258835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:39.258791 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:41.326931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:41.311868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:42.256410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:42.256379 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:29:43.252444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:43.252397 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:43.452807 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:43.452770 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:44.113929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.112977 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:44.270120 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.270072 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp"] Oct 29 22:29:44.338410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.338370 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp"] Oct 29 22:29:44.340448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.340422 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.387201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.387116 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.387792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.387752 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.390846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.388246 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.391199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.391179 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77rqb\" (UniqueName: \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-kube-api-access-77rqb\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.397599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.397564 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"agc-cred-d3185baddced\"" Oct 29 22:29:44.397786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.397754 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"agc-cred-702ebdff3648\"" Oct 29 22:29:44.397870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.397795 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"payload-processing-external-model-e2e--605ba260-dockercfg-56bp5\"" Oct 29 22:29:44.416940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.413554 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"routing-overlay\"" Oct 29 22:29:44.416940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.413738 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b9ed94\"" Oct 29 22:29:44.416940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.413883 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"openshift-service-ca.crt\"" Oct 29 22:29:44.416940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.413998 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b9ed94\"/\"kube-root-ca.crt\"" Oct 29 22:29:44.448927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.448806 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:44.493397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.493355 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.493607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.493534 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.493673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.493612 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.493738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.493717 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77rqb\" (UniqueName: \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-kube-api-access-77rqb\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.494424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.494394 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.494670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.494644 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.496882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.496855 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.531663 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.531629 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rqb\" (UniqueName: \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-kube-api-access-77rqb\") pod \"payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:44.691073 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:44.690983 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:45.453983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:45.453303 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp"] Oct 29 22:29:45.524716 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:45.524627 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" event={"ID":"c9929e56-05ff-4138-96af-66fd1d7b329f","Type":"ContainerStarted","Data":"7f7da758dcf034b582d3b7034d02d28601eda4e1af86115a0019db7052ac854d"} Oct 29 22:29:45.982405 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:29:45.982276 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:29:46.547635 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:46.547593 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" event={"ID":"c9929e56-05ff-4138-96af-66fd1d7b329f","Type":"ContainerStarted","Data":"92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211"} Oct 29 22:29:46.548225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:46.547745 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:46.593704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:46.593653 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" podStartSLOduration=2.593637927 podStartE2EDuration="2.593637927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:46.591833253 +0000 UTC m=+2852.066421670" watchObservedRunningTime="2026-10-29 22:29:46.593637927 +0000 UTC m=+2852.068226345" Oct 29 22:29:46.927289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:46.924613 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:48.931313 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:48.923346 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:50.570798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:50.570756 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:50.750846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:50.750805 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:50.914899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:50.914862 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:52.560534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:52.560494 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:54.271987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:54.271946 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:55.294501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:55.294457 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:55.478934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:55.478603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:56.497928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:56.489870 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:29:56.617024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:56.615115 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:29:56.694994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:56.694117 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:29:56.841285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:56.837387 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:29:56.893935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:56.893123 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:29:57.561162 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:57.561126 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:29:57.676397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:57.675454 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:59.516943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:29:59.516391 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:01.434927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:01.428039 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:03.171933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:03.165937 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:04.845998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:04.844564 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:06.639563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:06.639523 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:08.450849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:08.450568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:10.202692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:10.202651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:11.374318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:11.374278 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:11.959516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:11.959478 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:13.747535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:13.747234 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:15.607367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:15.607325 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:15.641467 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:15.641091 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:30:17.354928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:17.354143 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:19.108212 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:19.108167 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:20.850783 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:20.850740 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:22.599931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:22.596058 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:24.443897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:24.442694 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:26.176928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:26.154202 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:27.861935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:27.861871 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:29.652641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:29.652604 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:31.351111 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:31.351074 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:33.365107 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:33.365065 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:35.151927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:35.144488 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:30:35.958387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:35.958338 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:36.532012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:36.531972 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh"] Oct 29 22:30:36.532863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:36.532808 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" containerID="cri-o://3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e" gracePeriod=30 Oct 29 22:30:36.543393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:36.543349 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv"] Oct 29 22:30:36.543893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:36.543850 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" podUID="82c80abc-aa30-4ae0-b724-6dff35e30669" containerName="discovery" containerID="cri-o://c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211" gracePeriod=30 Oct 29 22:30:36.987187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:36.987153 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:30:37.036878 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.036838 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-kube-api-access-44rrg\" (UniqueName: \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-kube-api-access-44rrg\") pod \"82c80abc-aa30-4ae0-b724-6dff35e30669\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " Oct 29 22:30:37.037077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.036917 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-dns-cert\") pod \"82c80abc-aa30-4ae0-b724-6dff35e30669\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " Oct 29 22:30:37.037077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.036948 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-ca-configmap\") pod \"82c80abc-aa30-4ae0-b724-6dff35e30669\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " Oct 29 22:30:37.037077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.037046 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-kubeconfig\") pod \"82c80abc-aa30-4ae0-b724-6dff35e30669\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " Oct 29 22:30:37.037282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.037256 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-token\" (UniqueName: \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-token\") pod \"82c80abc-aa30-4ae0-b724-6dff35e30669\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " Oct 29 22:30:37.037413 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.037393 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82c80abc-aa30-4ae0-b724-6dff35e30669-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82c80abc-aa30-4ae0-b724-6dff35e30669-local-certs\") pod \"82c80abc-aa30-4ae0-b724-6dff35e30669\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " Oct 29 22:30:37.038464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.037746 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-cacerts\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-cacerts\") pod \"82c80abc-aa30-4ae0-b724-6dff35e30669\" (UID: \"82c80abc-aa30-4ae0-b724-6dff35e30669\") " Oct 29 22:30:37.038464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.038423 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-ca-configmap" pod "82c80abc-aa30-4ae0-b724-6dff35e30669" (UID: "82c80abc-aa30-4ae0-b724-6dff35e30669"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:37.038667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.038598 4952 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-ca-configmap\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:37.039870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.039832 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-dns-cert" pod "82c80abc-aa30-4ae0-b724-6dff35e30669" (UID: "82c80abc-aa30-4ae0-b724-6dff35e30669"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:37.040152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.040074 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-kube-api-access-44rrg" pod "82c80abc-aa30-4ae0-b724-6dff35e30669" (UID: "82c80abc-aa30-4ae0-b724-6dff35e30669"). InnerVolumeSpecName "kube-api-access-44rrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:37.041775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.041751 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-cacerts" pod "82c80abc-aa30-4ae0-b724-6dff35e30669" (UID: "82c80abc-aa30-4ae0-b724-6dff35e30669"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:37.042136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.042103 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c80abc-aa30-4ae0-b724-6dff35e30669-local-certs" pod "82c80abc-aa30-4ae0-b724-6dff35e30669" (UID: "82c80abc-aa30-4ae0-b724-6dff35e30669"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:37.042972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.042439 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-token" pod "82c80abc-aa30-4ae0-b724-6dff35e30669" (UID: "82c80abc-aa30-4ae0-b724-6dff35e30669"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:37.043325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.043299 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-kubeconfig" pod "82c80abc-aa30-4ae0-b724-6dff35e30669" (UID: "82c80abc-aa30-4ae0-b724-6dff35e30669"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:37.140986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.127125 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b"] Oct 29 22:30:37.145738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.142926 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44rrg\" (UniqueName: \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-kube-api-access-44rrg\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:37.145738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.142962 4952 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-csr-dns-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:37.145738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.142976 4952 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-kubeconfig\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:37.145738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.142988 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82c80abc-aa30-4ae0-b724-6dff35e30669-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:37.145738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.143000 4952 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82c80abc-aa30-4ae0-b724-6dff35e30669-local-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:37.145738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.143012 4952 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82c80abc-aa30-4ae0-b724-6dff35e30669-cacerts\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:37.145738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.144684 4952 cpu_manager.go:414] "containerMap: removing container" podUID="82c80abc-aa30-4ae0-b724-6dff35e30669" containerName="discovery" Oct 29 22:30:37.145738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.144711 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c80abc-aa30-4ae0-b724-6dff35e30669" containerName="discovery" Oct 29 22:30:37.145738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.144955 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c80abc-aa30-4ae0-b724-6dff35e30669" containerName="discovery" Oct 29 22:30:37.149665 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.149563 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx"] Oct 29 22:30:37.149820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.149725 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" Oct 29 22:30:37.159687 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.159155 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp"] Oct 29 22:30:37.161371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.160115 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" Oct 29 22:30:37.182982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.182106 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:30:37.186098 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.186075 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2"] Oct 29 22:30:37.196619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.196588 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" Oct 29 22:30:37.198200 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.198140 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq"] Oct 29 22:30:37.213600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.206416 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" Oct 29 22:30:37.236951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.227790 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2"] Oct 29 22:30:37.236951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.228272 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.244946 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d669ddd4-8a30-4078-9c55-870d0f8d948e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq\" (UID: \"d669ddd4-8a30-4078-9c55-870d0f8d948e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.244982 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjrf\" (UniqueName: \"kubernetes.io/projected/06ea33cb-020d-415a-b518-37018182741a-kube-api-access-wvjrf\") pod \"maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp\" (UID: \"06ea33cb-020d-415a-b518-37018182741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.245043 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df7b3a-e784-48ec-be58-f5620476c8f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2\" (UID: \"95df7b3a-e784-48ec-be58-f5620476c8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.245105 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19785bf3-8a8c-4346-9a06-01123ccf2378-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx\" (UID: \"19785bf3-8a8c-4346-9a06-01123ccf2378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.245134 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ea33cb-020d-415a-b518-37018182741a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp\" (UID: \"06ea33cb-020d-415a-b518-37018182741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.245162 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzdd\" (UniqueName: \"kubernetes.io/projected/19785bf3-8a8c-4346-9a06-01123ccf2378-kube-api-access-8dzdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx\" (UID: \"19785bf3-8a8c-4346-9a06-01123ccf2378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.245238 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg27c\" (UniqueName: \"kubernetes.io/projected/d669ddd4-8a30-4078-9c55-870d0f8d948e-kube-api-access-qg27c\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq\" (UID: \"d669ddd4-8a30-4078-9c55-870d0f8d948e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.245264 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjz78\" (UniqueName: \"kubernetes.io/projected/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-kube-api-access-hjz78\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b\" (UID: \"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.245311 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fss5q\" (UniqueName: \"kubernetes.io/projected/95df7b3a-e784-48ec-be58-f5620476c8f9-kube-api-access-fss5q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2\" (UID: \"95df7b3a-e784-48ec-be58-f5620476c8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.245331 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b\" (UID: \"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.251006 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh"] Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.251262 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" Oct 29 22:30:37.265929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.262293 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9"] Oct 29 22:30:37.266976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.266379 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" Oct 29 22:30:37.268988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.268182 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs"] Oct 29 22:30:37.270597 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.270568 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" Oct 29 22:30:37.280094 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.280067 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd"] Oct 29 22:30:37.280235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.280179 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" Oct 29 22:30:37.286432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.286407 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp"] Oct 29 22:30:37.286851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.286828 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" Oct 29 22:30:37.292507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.292482 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh"] Oct 29 22:30:37.293182 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.293158 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" Oct 29 22:30:37.298134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.298106 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5"] Oct 29 22:30:37.299960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.299878 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" Oct 29 22:30:37.304567 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.304548 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6"] Oct 29 22:30:37.304681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.304667 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" Oct 29 22:30:37.309695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.309674 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" Oct 29 22:30:37.310406 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.310388 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7"] Oct 29 22:30:37.316961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.315536 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" Oct 29 22:30:37.323767 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.323738 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6"] Oct 29 22:30:37.329744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.329722 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd"] Oct 29 22:30:37.329863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.329829 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" Oct 29 22:30:37.335788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.335763 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" Oct 29 22:30:37.336462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.336443 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f"] Oct 29 22:30:37.341704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.341683 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" Oct 29 22:30:37.344257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.344236 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b"] Oct 29 22:30:37.347467 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.347441 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d669ddd4-8a30-4078-9c55-870d0f8d948e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq\" (UID: \"d669ddd4-8a30-4078-9c55-870d0f8d948e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" Oct 29 22:30:37.347779 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.347755 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjrf\" (UniqueName: \"kubernetes.io/projected/06ea33cb-020d-415a-b518-37018182741a-kube-api-access-wvjrf\") pod \"maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp\" (UID: \"06ea33cb-020d-415a-b518-37018182741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" Oct 29 22:30:37.349954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.348196 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df7b3a-e784-48ec-be58-f5620476c8f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2\" (UID: \"95df7b3a-e784-48ec-be58-f5620476c8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" Oct 29 22:30:37.349954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.348292 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19785bf3-8a8c-4346-9a06-01123ccf2378-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx\" (UID: \"19785bf3-8a8c-4346-9a06-01123ccf2378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" Oct 29 22:30:37.349954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.348337 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ea33cb-020d-415a-b518-37018182741a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp\" (UID: \"06ea33cb-020d-415a-b518-37018182741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" Oct 29 22:30:37.349954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.348380 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzdd\" (UniqueName: \"kubernetes.io/projected/19785bf3-8a8c-4346-9a06-01123ccf2378-kube-api-access-8dzdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx\" (UID: \"19785bf3-8a8c-4346-9a06-01123ccf2378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" Oct 29 22:30:37.349954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.348496 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg27c\" (UniqueName: \"kubernetes.io/projected/d669ddd4-8a30-4078-9c55-870d0f8d948e-kube-api-access-qg27c\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq\" (UID: \"d669ddd4-8a30-4078-9c55-870d0f8d948e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" Oct 29 22:30:37.349954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.348535 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjz78\" (UniqueName: \"kubernetes.io/projected/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-kube-api-access-hjz78\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b\" (UID: \"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" Oct 29 22:30:37.349954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.348604 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fss5q\" (UniqueName: \"kubernetes.io/projected/95df7b3a-e784-48ec-be58-f5620476c8f9-kube-api-access-fss5q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2\" (UID: \"95df7b3a-e784-48ec-be58-f5620476c8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" Oct 29 22:30:37.349954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.348633 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b\" (UID: \"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" Oct 29 22:30:37.349954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.348916 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d669ddd4-8a30-4078-9c55-870d0f8d948e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq\" (UID: \"d669ddd4-8a30-4078-9c55-870d0f8d948e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" Oct 29 22:30:37.349954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.349417 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b\" (UID: \"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" Oct 29 22:30:37.351602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.349797 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ea33cb-020d-415a-b518-37018182741a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp\" (UID: \"06ea33cb-020d-415a-b518-37018182741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" Oct 29 22:30:37.365709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.360345 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx"] Oct 29 22:30:37.365709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.362808 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19785bf3-8a8c-4346-9a06-01123ccf2378-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx\" (UID: \"19785bf3-8a8c-4346-9a06-01123ccf2378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" Oct 29 22:30:37.367667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.367638 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df7b3a-e784-48ec-be58-f5620476c8f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2\" (UID: \"95df7b3a-e784-48ec-be58-f5620476c8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" Oct 29 22:30:37.374492 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.372760 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp"] Oct 29 22:30:37.379231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.379209 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjz78\" (UniqueName: \"kubernetes.io/projected/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-kube-api-access-hjz78\") pod \"maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b\" (UID: \"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" Oct 29 22:30:37.383156 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.383132 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2"] Oct 29 22:30:37.384079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.384049 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg27c\" (UniqueName: \"kubernetes.io/projected/d669ddd4-8a30-4078-9c55-870d0f8d948e-kube-api-access-qg27c\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq\" (UID: \"d669ddd4-8a30-4078-9c55-870d0f8d948e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" Oct 29 22:30:37.386358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.386327 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fss5q\" (UniqueName: \"kubernetes.io/projected/95df7b3a-e784-48ec-be58-f5620476c8f9-kube-api-access-fss5q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2\" (UID: \"95df7b3a-e784-48ec-be58-f5620476c8f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" Oct 29 22:30:37.391213 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.391154 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjrf\" (UniqueName: \"kubernetes.io/projected/06ea33cb-020d-415a-b518-37018182741a-kube-api-access-wvjrf\") pod \"maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp\" (UID: \"06ea33cb-020d-415a-b518-37018182741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" Oct 29 22:30:37.393478 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.393451 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzdd\" (UniqueName: \"kubernetes.io/projected/19785bf3-8a8c-4346-9a06-01123ccf2378-kube-api-access-8dzdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx\" (UID: \"19785bf3-8a8c-4346-9a06-01123ccf2378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" Oct 29 22:30:37.400102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.400078 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq"] Oct 29 22:30:37.409016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.408515 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2"] Oct 29 22:30:37.412811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.412787 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd"] Oct 29 22:30:37.417994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.417962 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6"] Oct 29 22:30:37.422405 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.422383 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5"] Oct 29 22:30:37.427813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.427767 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7"] Oct 29 22:30:37.432778 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.432756 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd"] Oct 29 22:30:37.438344 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.438323 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f"] Oct 29 22:30:37.443272 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.443254 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9"] Oct 29 22:30:37.449173 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.449151 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp"] Oct 29 22:30:37.455093 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.455074 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs"] Oct 29 22:30:37.458097 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458042 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e3163c-2722-49cb-a17d-11af94a932b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7\" (UID: \"56e3163c-2722-49cb-a17d-11af94a932b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" Oct 29 22:30:37.458194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458101 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e852730b-443d-4644-9a27-de432696d5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd\" (UID: \"e852730b-443d-4644-9a27-de432696d5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" Oct 29 22:30:37.458194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458148 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdfmz\" (UniqueName: \"kubernetes.io/projected/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-kube-api-access-tdfmz\") pod \"maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5\" (UID: \"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" Oct 29 22:30:37.458194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458196 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2\" (UID: \"8dc0b51a-0307-4c90-b5ca-cd00ea927a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" Oct 29 22:30:37.458361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458250 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh\" (UID: \"c0e57d7c-7064-43f7-bec3-b92ae3aba56e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" Oct 29 22:30:37.458361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458275 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwblc\" (UniqueName: \"kubernetes.io/projected/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-kube-api-access-nwblc\") pod \"maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh\" (UID: \"c0e57d7c-7064-43f7-bec3-b92ae3aba56e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" Oct 29 22:30:37.458361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458315 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd\" (UID: \"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" Oct 29 22:30:37.458361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458345 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk552\" (UniqueName: \"kubernetes.io/projected/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-kube-api-access-gk552\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2\" (UID: \"8dc0b51a-0307-4c90-b5ca-cd00ea927a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" Oct 29 22:30:37.458554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458380 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64j2x\" (UniqueName: \"kubernetes.io/projected/ae173ee2-aa93-41b3-a205-7bf63dfdba83-kube-api-access-64j2x\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6\" (UID: \"ae173ee2-aa93-41b3-a205-7bf63dfdba83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" Oct 29 22:30:37.458554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458451 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk8dk\" (UniqueName: \"kubernetes.io/projected/e852730b-443d-4644-9a27-de432696d5b9-kube-api-access-vk8dk\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd\" (UID: \"e852730b-443d-4644-9a27-de432696d5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" Oct 29 22:30:37.458554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458485 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9tj\" (UniqueName: \"kubernetes.io/projected/2412c3a1-19b8-4bba-8399-590859b54a7e-kube-api-access-2l9tj\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp\" (UID: \"2412c3a1-19b8-4bba-8399-590859b54a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" Oct 29 22:30:37.458660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458568 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz665\" (UniqueName: \"kubernetes.io/projected/4c674559-f3d7-47cf-a264-aa41039dc61f-kube-api-access-jz665\") pod \"maas-api-key-cleanup-29888550-hmrdh\" (UID: \"4c674559-f3d7-47cf-a264-aa41039dc61f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" Oct 29 22:30:37.458660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458611 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c674559-f3d7-47cf-a264-aa41039dc61f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888550-hmrdh\" (UID: \"4c674559-f3d7-47cf-a264-aa41039dc61f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" Oct 29 22:30:37.458660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458645 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2412c3a1-19b8-4bba-8399-590859b54a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp\" (UID: \"2412c3a1-19b8-4bba-8399-590859b54a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" Oct 29 22:30:37.458791 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.458768 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae173ee2-aa93-41b3-a205-7bf63dfdba83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6\" (UID: \"ae173ee2-aa93-41b3-a205-7bf63dfdba83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" Oct 29 22:30:37.459177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459143 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dafdb0e0-ba9f-4e05-ad91-1482995b0466-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs\" (UID: \"dafdb0e0-ba9f-4e05-ad91-1482995b0466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" Oct 29 22:30:37.459273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459250 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575c1d37-7dc0-4270-b023-192457eadac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6\" (UID: \"575c1d37-7dc0-4270-b023-192457eadac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" Oct 29 22:30:37.459339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459300 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d92b54-ecdc-4dc5-842b-be8811d0a761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f\" (UID: \"f9d92b54-ecdc-4dc5-842b-be8811d0a761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" Oct 29 22:30:37.459446 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459382 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7rr8\" (UniqueName: \"kubernetes.io/projected/f9d92b54-ecdc-4dc5-842b-be8811d0a761-kube-api-access-z7rr8\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f\" (UID: \"f9d92b54-ecdc-4dc5-842b-be8811d0a761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" Oct 29 22:30:37.459446 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459421 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ptdx\" (UniqueName: \"kubernetes.io/projected/575c1d37-7dc0-4270-b023-192457eadac8-kube-api-access-9ptdx\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6\" (UID: \"575c1d37-7dc0-4270-b023-192457eadac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" Oct 29 22:30:37.459562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459521 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rc2b\" (UniqueName: \"kubernetes.io/projected/56e3163c-2722-49cb-a17d-11af94a932b5-kube-api-access-6rc2b\") pod \"maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7\" (UID: \"56e3163c-2722-49cb-a17d-11af94a932b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" Oct 29 22:30:37.459619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459561 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b22f6\" (UniqueName: \"kubernetes.io/projected/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-kube-api-access-b22f6\") pod \"maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd\" (UID: \"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" Oct 29 22:30:37.459619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459587 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ft6c\" (UniqueName: \"kubernetes.io/projected/49f4055f-a3b2-4146-a50a-888fb5d73a35-kube-api-access-6ft6c\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9\" (UID: \"49f4055f-a3b2-4146-a50a-888fb5d73a35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" Oct 29 22:30:37.459746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459614 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f4055f-a3b2-4146-a50a-888fb5d73a35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9\" (UID: \"49f4055f-a3b2-4146-a50a-888fb5d73a35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" Oct 29 22:30:37.459746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459724 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbmq\" (UniqueName: \"kubernetes.io/projected/dafdb0e0-ba9f-4e05-ad91-1482995b0466-kube-api-access-qdbmq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs\" (UID: \"dafdb0e0-ba9f-4e05-ad91-1482995b0466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" Oct 29 22:30:37.459856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.459786 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5\" (UID: \"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" Oct 29 22:30:37.460293 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.460276 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh"] Oct 29 22:30:37.464389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.464368 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh"] Oct 29 22:30:37.468509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.468491 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6"] Oct 29 22:30:37.528708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.528672 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" Oct 29 22:30:37.539966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.539929 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" event={"ID":"82c80abc-aa30-4ae0-b724-6dff35e30669","Type":"ContainerDied","Data":"c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211"} Oct 29 22:30:37.539966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.539967 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" Oct 29 22:30:37.540668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.539992 4952 scope.go:122] "RemoveContainer" containerID="c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211" Oct 29 22:30:37.540668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.539895 4952 generic.go:356] "Generic (PLEG): container finished" podID="82c80abc-aa30-4ae0-b724-6dff35e30669" containerID="c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211" exitCode=0 Oct 29 22:30:37.540668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.540164 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv" event={"ID":"82c80abc-aa30-4ae0-b724-6dff35e30669","Type":"ContainerDied","Data":"f54ee598997779469e562ebdd08e90553b981ca1914e4d766bb686c2aa9800e1"} Oct 29 22:30:37.552124 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.552099 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" Oct 29 22:30:37.554068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.554050 4952 scope.go:122] "RemoveContainer" containerID="c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211" Oct 29 22:30:37.554378 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:37.554361 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211\": container with ID starting with c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211 not found: ID does not exist" containerID="c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211" Oct 29 22:30:37.554454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.554388 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211"} err="failed to get container status \"c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211\": rpc error: code = NotFound desc = could not find container \"c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211\": container with ID starting with c18026d45e5f19d9ef2a07b3ee1be61243a8acf1d323630e559475e065846211 not found: ID does not exist" Oct 29 22:30:37.561042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561012 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh\" (UID: \"c0e57d7c-7064-43f7-bec3-b92ae3aba56e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" Oct 29 22:30:37.561149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561056 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwblc\" (UniqueName: \"kubernetes.io/projected/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-kube-api-access-nwblc\") pod \"maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh\" (UID: \"c0e57d7c-7064-43f7-bec3-b92ae3aba56e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" Oct 29 22:30:37.561149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561096 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd\" (UID: \"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" Oct 29 22:30:37.561149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561127 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk552\" (UniqueName: \"kubernetes.io/projected/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-kube-api-access-gk552\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2\" (UID: \"8dc0b51a-0307-4c90-b5ca-cd00ea927a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" Oct 29 22:30:37.561350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561165 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64j2x\" (UniqueName: \"kubernetes.io/projected/ae173ee2-aa93-41b3-a205-7bf63dfdba83-kube-api-access-64j2x\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6\" (UID: \"ae173ee2-aa93-41b3-a205-7bf63dfdba83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" Oct 29 22:30:37.561350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561225 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk8dk\" (UniqueName: \"kubernetes.io/projected/e852730b-443d-4644-9a27-de432696d5b9-kube-api-access-vk8dk\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd\" (UID: \"e852730b-443d-4644-9a27-de432696d5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" Oct 29 22:30:37.561350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561257 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9tj\" (UniqueName: \"kubernetes.io/projected/2412c3a1-19b8-4bba-8399-590859b54a7e-kube-api-access-2l9tj\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp\" (UID: \"2412c3a1-19b8-4bba-8399-590859b54a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" Oct 29 22:30:37.561350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561303 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz665\" (UniqueName: \"kubernetes.io/projected/4c674559-f3d7-47cf-a264-aa41039dc61f-kube-api-access-jz665\") pod \"maas-api-key-cleanup-29888550-hmrdh\" (UID: \"4c674559-f3d7-47cf-a264-aa41039dc61f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" Oct 29 22:30:37.561586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561420 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c674559-f3d7-47cf-a264-aa41039dc61f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888550-hmrdh\" (UID: \"4c674559-f3d7-47cf-a264-aa41039dc61f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" Oct 29 22:30:37.561586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561464 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2412c3a1-19b8-4bba-8399-590859b54a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp\" (UID: \"2412c3a1-19b8-4bba-8399-590859b54a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" Oct 29 22:30:37.561586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561507 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae173ee2-aa93-41b3-a205-7bf63dfdba83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6\" (UID: \"ae173ee2-aa93-41b3-a205-7bf63dfdba83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" Oct 29 22:30:37.561586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561553 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dafdb0e0-ba9f-4e05-ad91-1482995b0466-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs\" (UID: \"dafdb0e0-ba9f-4e05-ad91-1482995b0466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" Oct 29 22:30:37.561884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561594 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575c1d37-7dc0-4270-b023-192457eadac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6\" (UID: \"575c1d37-7dc0-4270-b023-192457eadac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" Oct 29 22:30:37.561884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561631 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d92b54-ecdc-4dc5-842b-be8811d0a761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f\" (UID: \"f9d92b54-ecdc-4dc5-842b-be8811d0a761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" Oct 29 22:30:37.561884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561670 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7rr8\" (UniqueName: \"kubernetes.io/projected/f9d92b54-ecdc-4dc5-842b-be8811d0a761-kube-api-access-z7rr8\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f\" (UID: \"f9d92b54-ecdc-4dc5-842b-be8811d0a761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" Oct 29 22:30:37.561884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561712 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ptdx\" (UniqueName: \"kubernetes.io/projected/575c1d37-7dc0-4270-b023-192457eadac8-kube-api-access-9ptdx\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6\" (UID: \"575c1d37-7dc0-4270-b023-192457eadac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" Oct 29 22:30:37.561884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561773 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rc2b\" (UniqueName: \"kubernetes.io/projected/56e3163c-2722-49cb-a17d-11af94a932b5-kube-api-access-6rc2b\") pod \"maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7\" (UID: \"56e3163c-2722-49cb-a17d-11af94a932b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" Oct 29 22:30:37.561884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561820 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b22f6\" (UniqueName: \"kubernetes.io/projected/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-kube-api-access-b22f6\") pod \"maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd\" (UID: \"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" Oct 29 22:30:37.561884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561863 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ft6c\" (UniqueName: \"kubernetes.io/projected/49f4055f-a3b2-4146-a50a-888fb5d73a35-kube-api-access-6ft6c\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9\" (UID: \"49f4055f-a3b2-4146-a50a-888fb5d73a35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" Oct 29 22:30:37.562297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561927 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd\" (UID: \"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" Oct 29 22:30:37.562297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.561898 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f4055f-a3b2-4146-a50a-888fb5d73a35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9\" (UID: \"49f4055f-a3b2-4146-a50a-888fb5d73a35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" Oct 29 22:30:37.562297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.562101 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbmq\" (UniqueName: \"kubernetes.io/projected/dafdb0e0-ba9f-4e05-ad91-1482995b0466-kube-api-access-qdbmq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs\" (UID: \"dafdb0e0-ba9f-4e05-ad91-1482995b0466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" Oct 29 22:30:37.562297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.562183 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5\" (UID: \"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" Oct 29 22:30:37.562297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.562195 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c674559-f3d7-47cf-a264-aa41039dc61f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888550-hmrdh\" (UID: \"4c674559-f3d7-47cf-a264-aa41039dc61f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" Oct 29 22:30:37.562570 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.562335 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e3163c-2722-49cb-a17d-11af94a932b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7\" (UID: \"56e3163c-2722-49cb-a17d-11af94a932b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" Oct 29 22:30:37.562570 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.562394 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e852730b-443d-4644-9a27-de432696d5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd\" (UID: \"e852730b-443d-4644-9a27-de432696d5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" Oct 29 22:30:37.562570 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.562448 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdfmz\" (UniqueName: \"kubernetes.io/projected/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-kube-api-access-tdfmz\") pod \"maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5\" (UID: \"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" Oct 29 22:30:37.562570 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.562504 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2\" (UID: \"8dc0b51a-0307-4c90-b5ca-cd00ea927a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" Oct 29 22:30:37.562805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.562615 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f4055f-a3b2-4146-a50a-888fb5d73a35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9\" (UID: \"49f4055f-a3b2-4146-a50a-888fb5d73a35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" Oct 29 22:30:37.563015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.562979 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5\" (UID: \"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" Oct 29 22:30:37.563424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.563401 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d92b54-ecdc-4dc5-842b-be8811d0a761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f\" (UID: \"f9d92b54-ecdc-4dc5-842b-be8811d0a761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" Oct 29 22:30:37.563540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.563409 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2412c3a1-19b8-4bba-8399-590859b54a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp\" (UID: \"2412c3a1-19b8-4bba-8399-590859b54a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" Oct 29 22:30:37.563654 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.563636 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dafdb0e0-ba9f-4e05-ad91-1482995b0466-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs\" (UID: \"dafdb0e0-ba9f-4e05-ad91-1482995b0466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" Oct 29 22:30:37.564272 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.564244 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e3163c-2722-49cb-a17d-11af94a932b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7\" (UID: \"56e3163c-2722-49cb-a17d-11af94a932b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" Oct 29 22:30:37.564391 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.564358 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2\" (UID: \"8dc0b51a-0307-4c90-b5ca-cd00ea927a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" Oct 29 22:30:37.564505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.564462 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e852730b-443d-4644-9a27-de432696d5b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd\" (UID: \"e852730b-443d-4644-9a27-de432696d5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" Oct 29 22:30:37.564836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.564817 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh\" (UID: \"c0e57d7c-7064-43f7-bec3-b92ae3aba56e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" Oct 29 22:30:37.564836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.564826 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575c1d37-7dc0-4270-b023-192457eadac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6\" (UID: \"575c1d37-7dc0-4270-b023-192457eadac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" Oct 29 22:30:37.565482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.565460 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae173ee2-aa93-41b3-a205-7bf63dfdba83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6\" (UID: \"ae173ee2-aa93-41b3-a205-7bf63dfdba83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" Oct 29 22:30:37.598794 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.598557 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" Oct 29 22:30:37.600481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.600400 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbmq\" (UniqueName: \"kubernetes.io/projected/dafdb0e0-ba9f-4e05-ad91-1482995b0466-kube-api-access-qdbmq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs\" (UID: \"dafdb0e0-ba9f-4e05-ad91-1482995b0466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" Oct 29 22:30:37.601398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.601338 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz665\" (UniqueName: \"kubernetes.io/projected/4c674559-f3d7-47cf-a264-aa41039dc61f-kube-api-access-jz665\") pod \"maas-api-key-cleanup-29888550-hmrdh\" (UID: \"4c674559-f3d7-47cf-a264-aa41039dc61f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" Oct 29 22:30:37.604499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.604379 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk8dk\" (UniqueName: \"kubernetes.io/projected/e852730b-443d-4644-9a27-de432696d5b9-kube-api-access-vk8dk\") pod \"maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd\" (UID: \"e852730b-443d-4644-9a27-de432696d5b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" Oct 29 22:30:37.605357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.605308 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7rr8\" (UniqueName: \"kubernetes.io/projected/f9d92b54-ecdc-4dc5-842b-be8811d0a761-kube-api-access-z7rr8\") pod \"maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f\" (UID: \"f9d92b54-ecdc-4dc5-842b-be8811d0a761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" Oct 29 22:30:37.606228 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.606180 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdfmz\" (UniqueName: \"kubernetes.io/projected/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-kube-api-access-tdfmz\") pod \"maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5\" (UID: \"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" Oct 29 22:30:37.606322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.606272 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b22f6\" (UniqueName: \"kubernetes.io/projected/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-kube-api-access-b22f6\") pod \"maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd\" (UID: \"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" Oct 29 22:30:37.609334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.609308 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" Oct 29 22:30:37.610337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.610290 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9tj\" (UniqueName: \"kubernetes.io/projected/2412c3a1-19b8-4bba-8399-590859b54a7e-kube-api-access-2l9tj\") pod \"maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp\" (UID: \"2412c3a1-19b8-4bba-8399-590859b54a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" Oct 29 22:30:37.610442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.610330 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ft6c\" (UniqueName: \"kubernetes.io/projected/49f4055f-a3b2-4146-a50a-888fb5d73a35-kube-api-access-6ft6c\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9\" (UID: \"49f4055f-a3b2-4146-a50a-888fb5d73a35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" Oct 29 22:30:37.615362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.615322 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64j2x\" (UniqueName: \"kubernetes.io/projected/ae173ee2-aa93-41b3-a205-7bf63dfdba83-kube-api-access-64j2x\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6\" (UID: \"ae173ee2-aa93-41b3-a205-7bf63dfdba83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" Oct 29 22:30:37.623803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.618615 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ptdx\" (UniqueName: \"kubernetes.io/projected/575c1d37-7dc0-4270-b023-192457eadac8-kube-api-access-9ptdx\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6\" (UID: \"575c1d37-7dc0-4270-b023-192457eadac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" Oct 29 22:30:37.623803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.620054 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwblc\" (UniqueName: \"kubernetes.io/projected/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-kube-api-access-nwblc\") pod \"maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh\" (UID: \"c0e57d7c-7064-43f7-bec3-b92ae3aba56e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" Oct 29 22:30:37.623803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.622186 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" Oct 29 22:30:37.652111 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.641229 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" Oct 29 22:30:37.668066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.665130 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk552\" (UniqueName: \"kubernetes.io/projected/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-kube-api-access-gk552\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2\" (UID: \"8dc0b51a-0307-4c90-b5ca-cd00ea927a16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" Oct 29 22:30:37.668066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.667825 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" Oct 29 22:30:37.670810 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.670749 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" Oct 29 22:30:37.677540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.677309 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rc2b\" (UniqueName: \"kubernetes.io/projected/56e3163c-2722-49cb-a17d-11af94a932b5-kube-api-access-6rc2b\") pod \"maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7\" (UID: \"56e3163c-2722-49cb-a17d-11af94a932b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" Oct 29 22:30:37.724445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.724197 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" Oct 29 22:30:37.724853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.724809 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" Oct 29 22:30:37.725695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.725640 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" Oct 29 22:30:37.726319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.726300 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" Oct 29 22:30:37.748745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.747962 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" Oct 29 22:30:37.748745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.748406 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" Oct 29 22:30:37.757784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.757756 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" Oct 29 22:30:37.774821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.774788 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" Oct 29 22:30:37.780202 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.780141 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" Oct 29 22:30:37.932192 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:37.931964 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" Oct 29 22:30:38.139495 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:38.139366 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b"] Oct 29 22:30:38.355251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:38.354653 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx"] Oct 29 22:30:38.370301 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:38.369524 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:38.393081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:38.393011 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv"] Oct 29 22:30:38.458621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:38.458583 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rlhrv"] Oct 29 22:30:38.469306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:38.469276 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq"] Oct 29 22:30:38.469542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:38.469525 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2"] Oct 29 22:30:39.003360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.003237 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c80abc-aa30-4ae0-b724-6dff35e30669" path="/var/lib/kubelet/pods/82c80abc-aa30-4ae0-b724-6dff35e30669/volumes" Oct 29 22:30:39.006976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.006952 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv"] Oct 29 22:30:39.007738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.007710 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" event={"ID":"19785bf3-8a8c-4346-9a06-01123ccf2378","Type":"ContainerStarted","Data":"da994ecffc8013a95be077c0ea22bff1b6f2763e89e867e12d4202c405282e1d"} Oct 29 22:30:39.009779 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.007885 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" event={"ID":"95df7b3a-e784-48ec-be58-f5620476c8f9","Type":"ContainerStarted","Data":"91fcc01bec67188812a7aa259eee463552fcb2dfe93c6ad6f399d75b25cf10c1"} Oct 29 22:30:39.010229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.010184 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" event={"ID":"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a","Type":"ContainerStarted","Data":"bc6a1500ff41d13967a7c96f2ac533f45f54e27141bc202996d206546747dd22"} Oct 29 22:30:39.011954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.011882 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" event={"ID":"d669ddd4-8a30-4078-9c55-870d0f8d948e","Type":"ContainerStarted","Data":"151e7856ca966adc9aebe7e6ce1c818ec155bd163665e7f0f27889994e99efbd"} Oct 29 22:30:39.095650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.095576 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" containerID="cri-o://79809d795fc4dbe78db8e808d66b7fe263ca0f096d8d349a26b8006598b8a858" gracePeriod=30 Oct 29 22:30:39.120352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.120320 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd"] Oct 29 22:30:39.179304 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.179224 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs"] Oct 29 22:30:39.194328 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.187391 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp"] Oct 29 22:30:39.262475 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:30:39.262186 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ea33cb_020d_415a_b518_37018182741a.slice/crio-3592766599a3e939d8f4f3ac92a587809af47603541326b452d834be4a7f1c3b WatchSource:0}: Error finding container 3592766599a3e939d8f4f3ac92a587809af47603541326b452d834be4a7f1c3b: Status 404 returned error can't find the container with id 3592766599a3e939d8f4f3ac92a587809af47603541326b452d834be4a7f1c3b Oct 29 22:30:39.276276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.275792 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7"] Oct 29 22:30:39.640303 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.640209 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2"] Oct 29 22:30:39.742256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.741991 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh"] Oct 29 22:30:39.750712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.750685 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd"] Oct 29 22:30:39.757803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.757714 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5"] Oct 29 22:30:39.766479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.765477 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6"] Oct 29 22:30:39.792615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.792588 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f"] Oct 29 22:30:39.815290 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.815080 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6"] Oct 29 22:30:39.855351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.855304 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp"] Oct 29 22:30:39.864793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.863117 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh"] Oct 29 22:30:39.935813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:39.927164 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9"] Oct 29 22:30:40.035270 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.035176 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" event={"ID":"ae173ee2-aa93-41b3-a205-7bf63dfdba83","Type":"ContainerStarted","Data":"df2da63a8dd51ca49613853a36abb97cf8db0dd272aad7b832ec412f18b1325d"} Oct 29 22:30:40.038204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.037265 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" event={"ID":"56e3163c-2722-49cb-a17d-11af94a932b5","Type":"ContainerStarted","Data":"9bfc372c669737c74ceeeb4a712fdd9382621a17827330c4c86a67ea37c08124"} Oct 29 22:30:40.038204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.037320 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" event={"ID":"56e3163c-2722-49cb-a17d-11af94a932b5","Type":"ContainerStarted","Data":"7f2dfdef2c79fd0b782e1b8bf62b3b00ac4ba6d39c4e3c7feddb678e30b4a4d8"} Oct 29 22:30:40.039646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.039614 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" event={"ID":"19785bf3-8a8c-4346-9a06-01123ccf2378","Type":"ContainerStarted","Data":"a6fac33351ef4b2dd5a65e8e98c258b69ab8af5c12b56987ff35819949b7312a"} Oct 29 22:30:40.043123 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.043089 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" event={"ID":"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6","Type":"ContainerStarted","Data":"04374893b4aa467948327197411456907c674786e693ff21e2d408e2a2d6648f"} Oct 29 22:30:40.045164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.045139 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" event={"ID":"dafdb0e0-ba9f-4e05-ad91-1482995b0466","Type":"ContainerStarted","Data":"7a2c1a0730e32d71632ad698e322cd249e546ed1b143955df23d21094a13c9ec"} Oct 29 22:30:40.046819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.046792 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" event={"ID":"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c","Type":"ContainerStarted","Data":"b2b3618c1163c41f1b37a52ba9015cfa8da052a85875bf6ffc5eaa9f9c6e7abb"} Oct 29 22:30:40.048308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.048281 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" event={"ID":"c0e57d7c-7064-43f7-bec3-b92ae3aba56e","Type":"ContainerStarted","Data":"27c76fd655a0dc49e0513be476e7d1b058e30d06bff76f4e092ab57eae46b916"} Oct 29 22:30:40.049980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.049928 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" event={"ID":"4c674559-f3d7-47cf-a264-aa41039dc61f","Type":"ContainerStarted","Data":"9ce8d73092f26e8406d908898ee39dfff9e691a14e5abe63dc5b59492691007e"} Oct 29 22:30:40.053521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.052581 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" event={"ID":"95df7b3a-e784-48ec-be58-f5620476c8f9","Type":"ContainerStarted","Data":"ba7b76e5e24a09c0642505f150c74d038189c22acd61cef0af504fc2a3d363f2"} Oct 29 22:30:40.056856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.055982 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" event={"ID":"e852730b-443d-4644-9a27-de432696d5b9","Type":"ContainerStarted","Data":"558e2d5735280fdbabfc96e7ce6934492d348a4e1b51d53380b69aaed727e72f"} Oct 29 22:30:40.056856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.056031 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" event={"ID":"e852730b-443d-4644-9a27-de432696d5b9","Type":"ContainerStarted","Data":"a11edb7a324b393fa5a276fc1fa21b53997aac7e9abd762186ed77a50e30c689"} Oct 29 22:30:40.058798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.058772 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" event={"ID":"575c1d37-7dc0-4270-b023-192457eadac8","Type":"ContainerStarted","Data":"7ffcfe62d745e2b7f40b99228a47f7ba988828921c4fd09dcfc203af9914e843"} Oct 29 22:30:40.064085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.064057 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" event={"ID":"8dc0b51a-0307-4c90-b5ca-cd00ea927a16","Type":"ContainerStarted","Data":"628afa52d168ee7d7e8fad93cdf8311d579fe95957a53af6cee1b2e31923a124"} Oct 29 22:30:40.069963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.069227 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" event={"ID":"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a","Type":"ContainerStarted","Data":"ba277de12abb6626449876fff9940cd20ba130887866a439d01a78d13ce496dd"} Oct 29 22:30:40.072155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.072125 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" event={"ID":"f9d92b54-ecdc-4dc5-842b-be8811d0a761","Type":"ContainerStarted","Data":"57be6952df9daba0fa0cd286a4cf054e2180f9140fe81b7240081ad097311930"} Oct 29 22:30:40.073764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.073741 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" event={"ID":"06ea33cb-020d-415a-b518-37018182741a","Type":"ContainerStarted","Data":"2ddb3a18b199b530ed2077755fcc40d95f2aae5e83fab092216013fb361e0629"} Oct 29 22:30:40.073866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.073779 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" event={"ID":"06ea33cb-020d-415a-b518-37018182741a","Type":"ContainerStarted","Data":"3592766599a3e939d8f4f3ac92a587809af47603541326b452d834be4a7f1c3b"} Oct 29 22:30:40.075766 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.075742 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" event={"ID":"d669ddd4-8a30-4078-9c55-870d0f8d948e","Type":"ContainerStarted","Data":"895c28feae86bcc16a72de9e2622aa887b3f3eb9b46596ff10d8c79bffc95fea"} Oct 29 22:30:40.077370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.077345 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" event={"ID":"2412c3a1-19b8-4bba-8399-590859b54a7e","Type":"ContainerStarted","Data":"75292a480f3394c1da645c4b046e6c3e5cce9cdb77167ceb9f89045227274418"} Oct 29 22:30:40.082580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.082535 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" podStartSLOduration=40.082523303 podStartE2EDuration="40.082523303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:40.080266621 +0000 UTC m=+2905.554855040" watchObservedRunningTime="2026-10-29 22:30:40.082523303 +0000 UTC m=+2905.557111751" Oct 29 22:30:40.188987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.188932 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" podStartSLOduration=40.188918125 podStartE2EDuration="40.188918125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:40.142072567 +0000 UTC m=+2905.616661011" watchObservedRunningTime="2026-10-29 22:30:40.188918125 +0000 UTC m=+2905.663506543" Oct 29 22:30:40.191089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.190236 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" podStartSLOduration=40.190224864 podStartE2EDuration="40.190224864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:40.185156129 +0000 UTC m=+2905.659744559" watchObservedRunningTime="2026-10-29 22:30:40.190224864 +0000 UTC m=+2905.664813282" Oct 29 22:30:40.244491 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.244015 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" podStartSLOduration=40.243995697 podStartE2EDuration="40.243995697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:40.22917598 +0000 UTC m=+2905.703764398" watchObservedRunningTime="2026-10-29 22:30:40.243995697 +0000 UTC m=+2905.718584116" Oct 29 22:30:40.279957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.279877 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" podStartSLOduration=40.279860007 podStartE2EDuration="40.279860007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:40.273823088 +0000 UTC m=+2905.748411508" watchObservedRunningTime="2026-10-29 22:30:40.279860007 +0000 UTC m=+2905.754448429" Oct 29 22:30:40.319916 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.319834 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" podStartSLOduration=40.319814659 podStartE2EDuration="40.319814659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:40.31618569 +0000 UTC m=+2905.790774110" watchObservedRunningTime="2026-10-29 22:30:40.319814659 +0000 UTC m=+2905.794403077" Oct 29 22:30:40.368539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.368021 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" podStartSLOduration=40.367988827 podStartE2EDuration="40.367988827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:40.365820254 +0000 UTC m=+2905.840408673" watchObservedRunningTime="2026-10-29 22:30:40.367988827 +0000 UTC m=+2905.842577245" Oct 29 22:30:40.408687 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:40.408648 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:41.089998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.089950 4952 generic.go:356] "Generic (PLEG): container finished" podID="06ea33cb-020d-415a-b518-37018182741a" containerID="2ddb3a18b199b530ed2077755fcc40d95f2aae5e83fab092216013fb361e0629" exitCode=60 Oct 29 22:30:41.090474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.090068 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" event={"ID":"06ea33cb-020d-415a-b518-37018182741a","Type":"ContainerDied","Data":"2ddb3a18b199b530ed2077755fcc40d95f2aae5e83fab092216013fb361e0629"} Oct 29 22:30:41.090605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.090582 4952 scope.go:122] "RemoveContainer" containerID="2ddb3a18b199b530ed2077755fcc40d95f2aae5e83fab092216013fb361e0629" Oct 29 22:30:41.093319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.093231 4952 generic.go:356] "Generic (PLEG): container finished" podID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerID="895c28feae86bcc16a72de9e2622aa887b3f3eb9b46596ff10d8c79bffc95fea" exitCode=60 Oct 29 22:30:41.093394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.093310 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" event={"ID":"d669ddd4-8a30-4078-9c55-870d0f8d948e","Type":"ContainerDied","Data":"895c28feae86bcc16a72de9e2622aa887b3f3eb9b46596ff10d8c79bffc95fea"} Oct 29 22:30:41.093591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.093575 4952 scope.go:122] "RemoveContainer" containerID="895c28feae86bcc16a72de9e2622aa887b3f3eb9b46596ff10d8c79bffc95fea" Oct 29 22:30:41.095418 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.095393 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" event={"ID":"2412c3a1-19b8-4bba-8399-590859b54a7e","Type":"ContainerStarted","Data":"798b69b20f81bea52ebacb13c7db892cdb3ebf5d7c989b7da543d40e44b4a4de"} Oct 29 22:30:41.098677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.098653 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" event={"ID":"ae173ee2-aa93-41b3-a205-7bf63dfdba83","Type":"ContainerStarted","Data":"df71961e122b97959fc07f8eefe3089ff6da0cd13b31f827376865ed93298422"} Oct 29 22:30:41.109570 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.109546 4952 generic.go:356] "Generic (PLEG): container finished" podID="56e3163c-2722-49cb-a17d-11af94a932b5" containerID="9bfc372c669737c74ceeeb4a712fdd9382621a17827330c4c86a67ea37c08124" exitCode=60 Oct 29 22:30:41.109721 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.109633 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" event={"ID":"56e3163c-2722-49cb-a17d-11af94a932b5","Type":"ContainerDied","Data":"9bfc372c669737c74ceeeb4a712fdd9382621a17827330c4c86a67ea37c08124"} Oct 29 22:30:41.109979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.109958 4952 scope.go:122] "RemoveContainer" containerID="9bfc372c669737c74ceeeb4a712fdd9382621a17827330c4c86a67ea37c08124" Oct 29 22:30:41.111803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.111779 4952 generic.go:356] "Generic (PLEG): container finished" podID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerID="a6fac33351ef4b2dd5a65e8e98c258b69ab8af5c12b56987ff35819949b7312a" exitCode=60 Oct 29 22:30:41.112302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.112257 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" event={"ID":"19785bf3-8a8c-4346-9a06-01123ccf2378","Type":"ContainerDied","Data":"a6fac33351ef4b2dd5a65e8e98c258b69ab8af5c12b56987ff35819949b7312a"} Oct 29 22:30:41.112674 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.112508 4952 scope.go:122] "RemoveContainer" containerID="a6fac33351ef4b2dd5a65e8e98c258b69ab8af5c12b56987ff35819949b7312a" Oct 29 22:30:41.114451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.114425 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" event={"ID":"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6","Type":"ContainerStarted","Data":"90628b1d57f576e02fa0409e403a4eeb5ee446cdbbef182406e7fc691a242357"} Oct 29 22:30:41.139532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.138899 4952 generic.go:356] "Generic (PLEG): container finished" podID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerID="d477e716f1d86d3822eaf07c1f099c61d927dab930fb05dd4722b5843d809b68" exitCode=60 Oct 29 22:30:41.139532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.139075 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" event={"ID":"dafdb0e0-ba9f-4e05-ad91-1482995b0466","Type":"ContainerDied","Data":"d477e716f1d86d3822eaf07c1f099c61d927dab930fb05dd4722b5843d809b68"} Oct 29 22:30:41.142188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.139820 4952 scope.go:122] "RemoveContainer" containerID="d477e716f1d86d3822eaf07c1f099c61d927dab930fb05dd4722b5843d809b68" Oct 29 22:30:41.149004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.148972 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" event={"ID":"49f4055f-a3b2-4146-a50a-888fb5d73a35","Type":"ContainerStarted","Data":"3a1f921209eeb49ab5f3186225c63e60c7adee9d34773828d6b61d63a280da90"} Oct 29 22:30:41.154190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.154167 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" event={"ID":"c0e57d7c-7064-43f7-bec3-b92ae3aba56e","Type":"ContainerStarted","Data":"f16df72a14d2f0f155ef9e2e17dbde6d5dbaba8c27f546871da43059ec97c690"} Oct 29 22:30:41.155798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.155780 4952 generic.go:356] "Generic (PLEG): container finished" podID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerID="ba7b76e5e24a09c0642505f150c74d038189c22acd61cef0af504fc2a3d363f2" exitCode=60 Oct 29 22:30:41.156121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.156097 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" event={"ID":"95df7b3a-e784-48ec-be58-f5620476c8f9","Type":"ContainerDied","Data":"ba7b76e5e24a09c0642505f150c74d038189c22acd61cef0af504fc2a3d363f2"} Oct 29 22:30:41.156355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.156336 4952 scope.go:122] "RemoveContainer" containerID="ba7b76e5e24a09c0642505f150c74d038189c22acd61cef0af504fc2a3d363f2" Oct 29 22:30:41.160583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.160467 4952 generic.go:356] "Generic (PLEG): container finished" podID="e852730b-443d-4644-9a27-de432696d5b9" containerID="558e2d5735280fdbabfc96e7ce6934492d348a4e1b51d53380b69aaed727e72f" exitCode=60 Oct 29 22:30:41.160583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.160569 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" event={"ID":"e852730b-443d-4644-9a27-de432696d5b9","Type":"ContainerDied","Data":"558e2d5735280fdbabfc96e7ce6934492d348a4e1b51d53380b69aaed727e72f"} Oct 29 22:30:41.160822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.160803 4952 scope.go:122] "RemoveContainer" containerID="558e2d5735280fdbabfc96e7ce6934492d348a4e1b51d53380b69aaed727e72f" Oct 29 22:30:41.162658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.162638 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" event={"ID":"8dc0b51a-0307-4c90-b5ca-cd00ea927a16","Type":"ContainerStarted","Data":"e4bec9c747242a08a4784f3499d9fece9ec7cbceed50f1c0cdd7b2d2ca253b4c"} Oct 29 22:30:41.164417 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.164397 4952 generic.go:356] "Generic (PLEG): container finished" podID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerID="ba277de12abb6626449876fff9940cd20ba130887866a439d01a78d13ce496dd" exitCode=60 Oct 29 22:30:41.164507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.164485 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" event={"ID":"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a","Type":"ContainerDied","Data":"ba277de12abb6626449876fff9940cd20ba130887866a439d01a78d13ce496dd"} Oct 29 22:30:41.164713 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.164694 4952 scope.go:122] "RemoveContainer" containerID="ba277de12abb6626449876fff9940cd20ba130887866a439d01a78d13ce496dd" Oct 29 22:30:41.166219 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.166191 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" event={"ID":"f9d92b54-ecdc-4dc5-842b-be8811d0a761","Type":"ContainerStarted","Data":"571f88b95726b89f7097faec7acd14fd51b1c3569a827731002e7b4b81e04763"} Oct 29 22:30:41.233032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.232976 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" podStartSLOduration=41.232961542 podStartE2EDuration="41.232961542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:41.228098755 +0000 UTC m=+2906.702687180" watchObservedRunningTime="2026-10-29 22:30:41.232961542 +0000 UTC m=+2906.707549998" Oct 29 22:30:41.405743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.405688 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" podStartSLOduration=41.40566976 podStartE2EDuration="41.40566976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:41.401525129 +0000 UTC m=+2906.876113547" watchObservedRunningTime="2026-10-29 22:30:41.40566976 +0000 UTC m=+2906.880258182" Oct 29 22:30:41.449602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.449537 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" podStartSLOduration=41.449521052 podStartE2EDuration="41.449521052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:41.443562474 +0000 UTC m=+2906.918150893" watchObservedRunningTime="2026-10-29 22:30:41.449521052 +0000 UTC m=+2906.924109469" Oct 29 22:30:41.497468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.497412 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" podStartSLOduration=41.497398088 podStartE2EDuration="41.497398088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:41.492763045 +0000 UTC m=+2906.967351504" watchObservedRunningTime="2026-10-29 22:30:41.497398088 +0000 UTC m=+2906.971986507" Oct 29 22:30:41.629690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.629640 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" podStartSLOduration=41.629627476 podStartE2EDuration="41.629627476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:41.629253737 +0000 UTC m=+2907.103842156" watchObservedRunningTime="2026-10-29 22:30:41.629627476 +0000 UTC m=+2907.104215894" Oct 29 22:30:41.717691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:41.717636 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" podStartSLOduration=41.717620251 podStartE2EDuration="41.717620251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:41.714701921 +0000 UTC m=+2907.189290339" watchObservedRunningTime="2026-10-29 22:30:41.717620251 +0000 UTC m=+2907.192208669" Oct 29 22:30:42.003518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.003493 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:30:42.068174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.068132 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-istio-token\" (UniqueName: \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-istio-token\") pod \"b03ee8fc-3424-4232-85ad-96a512607f60\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " Oct 29 22:30:42.068871 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.068845 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-socket\") pod \"b03ee8fc-3424-4232-85ad-96a512607f60\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " Oct 29 22:30:42.069485 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.069465 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-certs\") pod \"b03ee8fc-3424-4232-85ad-96a512607f60\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " Oct 29 22:30:42.069899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.069879 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03ee8fc-3424-4232-85ad-96a512607f60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03ee8fc-3424-4232-85ad-96a512607f60-istiod-ca-cert\") pod \"b03ee8fc-3424-4232-85ad-96a512607f60\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " Oct 29 22:30:42.070427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.070410 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-credential-socket\") pod \"b03ee8fc-3424-4232-85ad-96a512607f60\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " Oct 29 22:30:42.070991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.070972 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b03ee8fc-3424-4232-85ad-96a512607f60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03ee8fc-3424-4232-85ad-96a512607f60-istio-podinfo\") pod \"b03ee8fc-3424-4232-85ad-96a512607f60\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " Oct 29 22:30:42.071499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.071481 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-kube-api-access-9bcfs\" (UniqueName: \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-kube-api-access-9bcfs\") pod \"b03ee8fc-3424-4232-85ad-96a512607f60\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " Oct 29 22:30:42.072021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.072001 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-data\") pod \"b03ee8fc-3424-4232-85ad-96a512607f60\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " Oct 29 22:30:42.072668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.069372 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-socket" pod "b03ee8fc-3424-4232-85ad-96a512607f60" (UID: "b03ee8fc-3424-4232-85ad-96a512607f60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:42.072844 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.069795 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-certs" pod "b03ee8fc-3424-4232-85ad-96a512607f60" (UID: "b03ee8fc-3424-4232-85ad-96a512607f60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:42.073044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.070325 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03ee8fc-3424-4232-85ad-96a512607f60-istiod-ca-cert" pod "b03ee8fc-3424-4232-85ad-96a512607f60" (UID: "b03ee8fc-3424-4232-85ad-96a512607f60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:42.073044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.070769 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-credential-socket" pod "b03ee8fc-3424-4232-85ad-96a512607f60" (UID: "b03ee8fc-3424-4232-85ad-96a512607f60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:42.073044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.072415 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-data" pod "b03ee8fc-3424-4232-85ad-96a512607f60" (UID: "b03ee8fc-3424-4232-85ad-96a512607f60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:42.073955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.073888 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.074085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.074071 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.074169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.074159 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03ee8fc-3424-4232-85ad-96a512607f60-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.074253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.074241 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.074329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.074318 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.131950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.131874 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b03ee8fc-3424-4232-85ad-96a512607f60-istio-podinfo" pod "b03ee8fc-3424-4232-85ad-96a512607f60" (UID: "b03ee8fc-3424-4232-85ad-96a512607f60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:30:42.132412 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.131975 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-istio-token" pod "b03ee8fc-3424-4232-85ad-96a512607f60" (UID: "b03ee8fc-3424-4232-85ad-96a512607f60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:42.176026 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.175989 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-envoy\") pod \"b03ee8fc-3424-4232-85ad-96a512607f60\" (UID: \"b03ee8fc-3424-4232-85ad-96a512607f60\") " Oct 29 22:30:42.176729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.176703 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03ee8fc-3424-4232-85ad-96a512607f60-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.176729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.176730 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.179056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.179023 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" event={"ID":"06ea33cb-020d-415a-b518-37018182741a","Type":"ContainerStarted","Data":"2f1884cfb0fb3388c0652d56e1c31b9d078825c6cac7c8baccde21b4a86ba805"} Oct 29 22:30:42.181323 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.180883 4952 generic.go:356] "Generic (PLEG): container finished" podID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerID="798b69b20f81bea52ebacb13c7db892cdb3ebf5d7c989b7da543d40e44b4a4de" exitCode=60 Oct 29 22:30:42.181323 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.181029 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" event={"ID":"2412c3a1-19b8-4bba-8399-590859b54a7e","Type":"ContainerDied","Data":"798b69b20f81bea52ebacb13c7db892cdb3ebf5d7c989b7da543d40e44b4a4de"} Oct 29 22:30:42.181323 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.181245 4952 scope.go:122] "RemoveContainer" containerID="798b69b20f81bea52ebacb13c7db892cdb3ebf5d7c989b7da543d40e44b4a4de" Oct 29 22:30:42.183237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.183215 4952 generic.go:356] "Generic (PLEG): container finished" podID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerID="df71961e122b97959fc07f8eefe3089ff6da0cd13b31f827376865ed93298422" exitCode=60 Oct 29 22:30:42.183332 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.183310 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" event={"ID":"ae173ee2-aa93-41b3-a205-7bf63dfdba83","Type":"ContainerDied","Data":"df71961e122b97959fc07f8eefe3089ff6da0cd13b31f827376865ed93298422"} Oct 29 22:30:42.183557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.183539 4952 scope.go:122] "RemoveContainer" containerID="df71961e122b97959fc07f8eefe3089ff6da0cd13b31f827376865ed93298422" Oct 29 22:30:42.185681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.185650 4952 generic.go:356] "Generic (PLEG): container finished" podID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerID="90628b1d57f576e02fa0409e403a4eeb5ee446cdbbef182406e7fc691a242357" exitCode=60 Oct 29 22:30:42.185805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.185788 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" event={"ID":"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6","Type":"ContainerDied","Data":"90628b1d57f576e02fa0409e403a4eeb5ee446cdbbef182406e7fc691a242357"} Oct 29 22:30:42.186190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.186170 4952 scope.go:122] "RemoveContainer" containerID="90628b1d57f576e02fa0409e403a4eeb5ee446cdbbef182406e7fc691a242357" Oct 29 22:30:42.188245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.188216 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" event={"ID":"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c","Type":"ContainerStarted","Data":"dd3f51905986ae828d964fe0d6c209531d7aca5ed4995bd0f76fa0b6b585f20d"} Oct 29 22:30:42.189853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.189803 4952 generic.go:356] "Generic (PLEG): container finished" podID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerID="dcae8b178c7e3c2e7a4d3435532b3d4390ed15d26363fd89f004a0eca4ae7e91" exitCode=60 Oct 29 22:30:42.190011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.189874 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" event={"ID":"49f4055f-a3b2-4146-a50a-888fb5d73a35","Type":"ContainerDied","Data":"dcae8b178c7e3c2e7a4d3435532b3d4390ed15d26363fd89f004a0eca4ae7e91"} Oct 29 22:30:42.190183 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.190152 4952 scope.go:122] "RemoveContainer" containerID="dcae8b178c7e3c2e7a4d3435532b3d4390ed15d26363fd89f004a0eca4ae7e91" Oct 29 22:30:42.191687 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.191667 4952 generic.go:356] "Generic (PLEG): container finished" podID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerID="f16df72a14d2f0f155ef9e2e17dbde6d5dbaba8c27f546871da43059ec97c690" exitCode=60 Oct 29 22:30:42.191775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.191726 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" event={"ID":"c0e57d7c-7064-43f7-bec3-b92ae3aba56e","Type":"ContainerDied","Data":"f16df72a14d2f0f155ef9e2e17dbde6d5dbaba8c27f546871da43059ec97c690"} Oct 29 22:30:42.192017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.191989 4952 scope.go:122] "RemoveContainer" containerID="f16df72a14d2f0f155ef9e2e17dbde6d5dbaba8c27f546871da43059ec97c690" Oct 29 22:30:42.193648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.193624 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" event={"ID":"4c674559-f3d7-47cf-a264-aa41039dc61f","Type":"ContainerStarted","Data":"4d9a22e83057871b137f689af259beb519544d2f046b251726bef737d3a8bd3f"} Oct 29 22:30:42.195776 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.195756 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" event={"ID":"95df7b3a-e784-48ec-be58-f5620476c8f9","Type":"ContainerStarted","Data":"34aca47fc2d12fc113a80dfe3762355dc4f025c61d4828658396366d8773854f"} Oct 29 22:30:42.205438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.204941 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" event={"ID":"575c1d37-7dc0-4270-b023-192457eadac8","Type":"ContainerStarted","Data":"5b40a2680558a445a69347c2a20a62602415139b15495ce1a5db42748ab54a1d"} Oct 29 22:30:42.207085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.207058 4952 generic.go:356] "Generic (PLEG): container finished" podID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerID="e4bec9c747242a08a4784f3499d9fece9ec7cbceed50f1c0cdd7b2d2ca253b4c" exitCode=60 Oct 29 22:30:42.207207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.207138 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" event={"ID":"8dc0b51a-0307-4c90-b5ca-cd00ea927a16","Type":"ContainerDied","Data":"e4bec9c747242a08a4784f3499d9fece9ec7cbceed50f1c0cdd7b2d2ca253b4c"} Oct 29 22:30:42.207335 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.207317 4952 scope.go:122] "RemoveContainer" containerID="e4bec9c747242a08a4784f3499d9fece9ec7cbceed50f1c0cdd7b2d2ca253b4c" Oct 29 22:30:42.209388 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.209362 4952 generic.go:356] "Generic (PLEG): container finished" podID="b03ee8fc-3424-4232-85ad-96a512607f60" containerID="3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e" exitCode=0 Oct 29 22:30:42.209511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.209487 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:30:42.209638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.209610 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" event={"ID":"b03ee8fc-3424-4232-85ad-96a512607f60","Type":"ContainerDied","Data":"3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e"} Oct 29 22:30:42.209717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.209646 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" event={"ID":"b03ee8fc-3424-4232-85ad-96a512607f60","Type":"ContainerDied","Data":"32c901b3fe1f77c67a1f7de0a07e476ac56dac3b0bcc0fdeb11c66d30bd798e0"} Oct 29 22:30:42.209717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.209674 4952 scope.go:122] "RemoveContainer" containerID="3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e" Oct 29 22:30:42.211778 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.211702 4952 generic.go:356] "Generic (PLEG): container finished" podID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerID="571f88b95726b89f7097faec7acd14fd51b1c3569a827731002e7b4b81e04763" exitCode=60 Oct 29 22:30:42.211863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.211808 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" event={"ID":"f9d92b54-ecdc-4dc5-842b-be8811d0a761","Type":"ContainerDied","Data":"571f88b95726b89f7097faec7acd14fd51b1c3569a827731002e7b4b81e04763"} Oct 29 22:30:42.213009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.212243 4952 scope.go:122] "RemoveContainer" containerID="571f88b95726b89f7097faec7acd14fd51b1c3569a827731002e7b4b81e04763" Oct 29 22:30:42.225616 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.225564 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-kube-api-access-9bcfs" pod "b03ee8fc-3424-4232-85ad-96a512607f60" (UID: "b03ee8fc-3424-4232-85ad-96a512607f60"). InnerVolumeSpecName "kube-api-access-9bcfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:42.325200 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.286899 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bcfs\" (UniqueName: \"kubernetes.io/projected/b03ee8fc-3424-4232-85ad-96a512607f60-kube-api-access-9bcfs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.388717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.388429 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-envoy" pod "b03ee8fc-3424-4232-85ad-96a512607f60" (UID: "b03ee8fc-3424-4232-85ad-96a512607f60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:42.390855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.390760 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:42.441608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.441520 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03ee8fc-3424-4232-85ad-96a512607f60-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:42.443168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.443121 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" podStartSLOduration=42.443106577 podStartE2EDuration="42.443106577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:42.438886339 +0000 UTC m=+2907.913474780" watchObservedRunningTime="2026-10-29 22:30:42.443106577 +0000 UTC m=+2907.917694995" Oct 29 22:30:42.589369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.589324 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" podStartSLOduration=42.589309224 podStartE2EDuration="42.589309224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:42.53362051 +0000 UTC m=+2908.008208929" watchObservedRunningTime="2026-10-29 22:30:42.589309224 +0000 UTC m=+2908.063897643" Oct 29 22:30:42.737975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.734593 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" podStartSLOduration=42.73457906 podStartE2EDuration="42.73457906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:42.734463504 +0000 UTC m=+2908.209051923" watchObservedRunningTime="2026-10-29 22:30:42.73457906 +0000 UTC m=+2908.209167479" Oct 29 22:30:42.895872 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.895733 4952 scope.go:122] "RemoveContainer" containerID="3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e" Oct 29 22:30:42.896224 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:42.896160 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e\": container with ID starting with 3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e not found: ID does not exist" containerID="3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e" Oct 29 22:30:42.896306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:42.896235 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e"} err="failed to get container status \"3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e\": rpc error: code = NotFound desc = could not find container \"3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e\": container with ID starting with 3c9ae9a52fb077f968c569bb5c418437a23d85c42eb2a72efbed718497d5c23e not found: ID does not exist" Oct 29 22:30:43.223568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.223533 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" event={"ID":"dafdb0e0-ba9f-4e05-ad91-1482995b0466","Type":"ContainerStarted","Data":"d02b270fa0d9aa2f16e67342c358fb48b4ae5a2604088f6c75371a114616a97e"} Oct 29 22:30:43.226722 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.226695 4952 generic.go:356] "Generic (PLEG): container finished" podID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerID="dd3f51905986ae828d964fe0d6c209531d7aca5ed4995bd0f76fa0b6b585f20d" exitCode=60 Oct 29 22:30:43.226882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.226770 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" event={"ID":"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c","Type":"ContainerDied","Data":"dd3f51905986ae828d964fe0d6c209531d7aca5ed4995bd0f76fa0b6b585f20d"} Oct 29 22:30:43.227115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.227096 4952 scope.go:122] "RemoveContainer" containerID="dd3f51905986ae828d964fe0d6c209531d7aca5ed4995bd0f76fa0b6b585f20d" Oct 29 22:30:43.228803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.228775 4952 generic.go:356] "Generic (PLEG): container finished" podID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerID="4d9a22e83057871b137f689af259beb519544d2f046b251726bef737d3a8bd3f" exitCode=60 Oct 29 22:30:43.228958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.228883 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" event={"ID":"4c674559-f3d7-47cf-a264-aa41039dc61f","Type":"ContainerDied","Data":"4d9a22e83057871b137f689af259beb519544d2f046b251726bef737d3a8bd3f"} Oct 29 22:30:43.229217 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.229176 4952 scope.go:122] "RemoveContainer" containerID="4d9a22e83057871b137f689af259beb519544d2f046b251726bef737d3a8bd3f" Oct 29 22:30:43.232030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.231547 4952 generic.go:356] "Generic (PLEG): container finished" podID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerID="34aca47fc2d12fc113a80dfe3762355dc4f025c61d4828658396366d8773854f" exitCode=60 Oct 29 22:30:43.232030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.231636 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" event={"ID":"95df7b3a-e784-48ec-be58-f5620476c8f9","Type":"ContainerDied","Data":"34aca47fc2d12fc113a80dfe3762355dc4f025c61d4828658396366d8773854f"} Oct 29 22:30:43.232030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.231726 4952 scope.go:122] "RemoveContainer" containerID="ba7b76e5e24a09c0642505f150c74d038189c22acd61cef0af504fc2a3d363f2" Oct 29 22:30:43.233688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.233656 4952 scope.go:122] "RemoveContainer" containerID="34aca47fc2d12fc113a80dfe3762355dc4f025c61d4828658396366d8773854f" Oct 29 22:30:43.242999 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:43.240259 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2_odh-ai-gateway-infra(95df7b3a-e784-48ec-be58-f5620476c8f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" Oct 29 22:30:43.255190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.254854 4952 generic.go:356] "Generic (PLEG): container finished" podID="e852730b-443d-4644-9a27-de432696d5b9" containerID="92645b29c9f68c6b37ddeeb06d7f86d5a08d7234c3ba680682eaf05c6754fdac" exitCode=60 Oct 29 22:30:43.255190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.254994 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" event={"ID":"e852730b-443d-4644-9a27-de432696d5b9","Type":"ContainerDied","Data":"92645b29c9f68c6b37ddeeb06d7f86d5a08d7234c3ba680682eaf05c6754fdac"} Oct 29 22:30:43.258211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.258184 4952 generic.go:356] "Generic (PLEG): container finished" podID="575c1d37-7dc0-4270-b023-192457eadac8" containerID="5b40a2680558a445a69347c2a20a62602415139b15495ce1a5db42748ab54a1d" exitCode=60 Oct 29 22:30:43.258441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.258422 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" event={"ID":"575c1d37-7dc0-4270-b023-192457eadac8","Type":"ContainerDied","Data":"5b40a2680558a445a69347c2a20a62602415139b15495ce1a5db42748ab54a1d"} Oct 29 22:30:43.259129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.259110 4952 scope.go:122] "RemoveContainer" containerID="92645b29c9f68c6b37ddeeb06d7f86d5a08d7234c3ba680682eaf05c6754fdac" Oct 29 22:30:43.259684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.259657 4952 scope.go:122] "RemoveContainer" containerID="5b40a2680558a445a69347c2a20a62602415139b15495ce1a5db42748ab54a1d" Oct 29 22:30:43.260728 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:43.259864 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd_odh-ai-gateway-infra(e852730b-443d-4644-9a27-de432696d5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" podUID="e852730b-443d-4644-9a27-de432696d5b9" Oct 29 22:30:43.261663 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.261637 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" event={"ID":"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a","Type":"ContainerStarted","Data":"0a5402ab405d8828d08a5349d318a983897893fe0962af9b354427a80cbe1bef"} Oct 29 22:30:43.288401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.288368 4952 generic.go:356] "Generic (PLEG): container finished" podID="06ea33cb-020d-415a-b518-37018182741a" containerID="2f1884cfb0fb3388c0652d56e1c31b9d078825c6cac7c8baccde21b4a86ba805" exitCode=60 Oct 29 22:30:43.288508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.288433 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" event={"ID":"06ea33cb-020d-415a-b518-37018182741a","Type":"ContainerDied","Data":"2f1884cfb0fb3388c0652d56e1c31b9d078825c6cac7c8baccde21b4a86ba805"} Oct 29 22:30:43.288965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.288938 4952 scope.go:122] "RemoveContainer" containerID="2f1884cfb0fb3388c0652d56e1c31b9d078825c6cac7c8baccde21b4a86ba805" Oct 29 22:30:43.289180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.288897 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" podStartSLOduration=43.288879912 podStartE2EDuration="43.288879912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:43.269083524 +0000 UTC m=+2908.743671944" watchObservedRunningTime="2026-10-29 22:30:43.288879912 +0000 UTC m=+2908.763468331" Oct 29 22:30:43.289650 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:43.289623 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp_odh-ai-gateway-infra(06ea33cb-020d-415a-b518-37018182741a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" podUID="06ea33cb-020d-415a-b518-37018182741a" Oct 29 22:30:43.292425 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.292405 4952 generic.go:356] "Generic (PLEG): container finished" podID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerID="c68d7bdfa30fac7bed79500c97e2bcc5075dcaeb190b6ad8923b41e188145359" exitCode=60 Oct 29 22:30:43.292634 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.292536 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" event={"ID":"d669ddd4-8a30-4078-9c55-870d0f8d948e","Type":"ContainerDied","Data":"c68d7bdfa30fac7bed79500c97e2bcc5075dcaeb190b6ad8923b41e188145359"} Oct 29 22:30:43.292947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.292887 4952 scope.go:122] "RemoveContainer" containerID="c68d7bdfa30fac7bed79500c97e2bcc5075dcaeb190b6ad8923b41e188145359" Oct 29 22:30:43.293362 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:43.293336 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq_odh-ai-gateway-infra(d669ddd4-8a30-4078-9c55-870d0f8d948e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" Oct 29 22:30:43.294601 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.294578 4952 generic.go:356] "Generic (PLEG): container finished" podID="56e3163c-2722-49cb-a17d-11af94a932b5" containerID="5250106705e37da31151211561a16dedf7e81d6814d01f0f08771cbf36725273" exitCode=60 Oct 29 22:30:43.294676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.294641 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" event={"ID":"56e3163c-2722-49cb-a17d-11af94a932b5","Type":"ContainerDied","Data":"5250106705e37da31151211561a16dedf7e81d6814d01f0f08771cbf36725273"} Oct 29 22:30:43.295042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.295025 4952 scope.go:122] "RemoveContainer" containerID="5250106705e37da31151211561a16dedf7e81d6814d01f0f08771cbf36725273" Oct 29 22:30:43.295311 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:43.295290 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7_odh-ai-gateway-infra(56e3163c-2722-49cb-a17d-11af94a932b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" Oct 29 22:30:43.297200 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.297176 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" event={"ID":"19785bf3-8a8c-4346-9a06-01123ccf2378","Type":"ContainerStarted","Data":"fcc03dfac458137483b9d19f6ece49a9e9dfc07d32487c5bd1cbe93dcb19e39f"} Oct 29 22:30:43.297441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.297425 4952 scope.go:122] "RemoveContainer" containerID="fcc03dfac458137483b9d19f6ece49a9e9dfc07d32487c5bd1cbe93dcb19e39f" Oct 29 22:30:43.297701 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:43.297681 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx_odh-ai-gateway-infra(19785bf3-8a8c-4346-9a06-01123ccf2378)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" Oct 29 22:30:43.319560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.319538 4952 scope.go:122] "RemoveContainer" containerID="558e2d5735280fdbabfc96e7ce6934492d348a4e1b51d53380b69aaed727e72f" Oct 29 22:30:43.499650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.499621 4952 scope.go:122] "RemoveContainer" containerID="2ddb3a18b199b530ed2077755fcc40d95f2aae5e83fab092216013fb361e0629" Oct 29 22:30:43.517697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.517673 4952 scope.go:122] "RemoveContainer" containerID="895c28feae86bcc16a72de9e2622aa887b3f3eb9b46596ff10d8c79bffc95fea" Oct 29 22:30:43.561198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:43.561093 4952 scope.go:122] "RemoveContainer" containerID="9bfc372c669737c74ceeeb4a712fdd9382621a17827330c4c86a67ea37c08124" Oct 29 22:30:44.309316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.309285 4952 generic.go:356] "Generic (PLEG): container finished" podID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerID="79809d795fc4dbe78db8e808d66b7fe263ca0f096d8d349a26b8006598b8a858" exitCode=0 Oct 29 22:30:44.309666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.309358 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" event={"ID":"a4d9b425-0133-4d70-a947-7926b68bc0e1","Type":"ContainerDied","Data":"79809d795fc4dbe78db8e808d66b7fe263ca0f096d8d349a26b8006598b8a858"} Oct 29 22:30:44.311559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.311540 4952 scope.go:122] "RemoveContainer" containerID="2f1884cfb0fb3388c0652d56e1c31b9d078825c6cac7c8baccde21b4a86ba805" Oct 29 22:30:44.311996 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.311968 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp_odh-ai-gateway-infra(06ea33cb-020d-415a-b518-37018182741a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" podUID="06ea33cb-020d-415a-b518-37018182741a" Oct 29 22:30:44.313297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.313278 4952 scope.go:122] "RemoveContainer" containerID="c68d7bdfa30fac7bed79500c97e2bcc5075dcaeb190b6ad8923b41e188145359" Oct 29 22:30:44.313670 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.313643 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq_odh-ai-gateway-infra(d669ddd4-8a30-4078-9c55-870d0f8d948e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" Oct 29 22:30:44.314751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.314729 4952 generic.go:356] "Generic (PLEG): container finished" podID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerID="bb504afd4934409689149ffcea778a4ea69d427277c0854a21b47469d248c844" exitCode=60 Oct 29 22:30:44.314946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.314793 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" event={"ID":"2412c3a1-19b8-4bba-8399-590859b54a7e","Type":"ContainerDied","Data":"bb504afd4934409689149ffcea778a4ea69d427277c0854a21b47469d248c844"} Oct 29 22:30:44.314946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.314852 4952 scope.go:122] "RemoveContainer" containerID="798b69b20f81bea52ebacb13c7db892cdb3ebf5d7c989b7da543d40e44b4a4de" Oct 29 22:30:44.315201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.315182 4952 scope.go:122] "RemoveContainer" containerID="bb504afd4934409689149ffcea778a4ea69d427277c0854a21b47469d248c844" Oct 29 22:30:44.315562 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.315527 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp_odh-ai-gateway-infra(2412c3a1-19b8-4bba-8399-590859b54a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" Oct 29 22:30:44.317031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.317008 4952 generic.go:356] "Generic (PLEG): container finished" podID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerID="f2fa6c40080e8f17796ebeb809674746cf234c52fec2307df9891102335fc391" exitCode=60 Oct 29 22:30:44.317139 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.317095 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" event={"ID":"ae173ee2-aa93-41b3-a205-7bf63dfdba83","Type":"ContainerDied","Data":"f2fa6c40080e8f17796ebeb809674746cf234c52fec2307df9891102335fc391"} Oct 29 22:30:44.317300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.317286 4952 scope.go:122] "RemoveContainer" containerID="f2fa6c40080e8f17796ebeb809674746cf234c52fec2307df9891102335fc391" Oct 29 22:30:44.317610 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.317587 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6_odh-ai-gateway-infra(ae173ee2-aa93-41b3-a205-7bf63dfdba83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" Oct 29 22:30:44.319018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.319000 4952 scope.go:122] "RemoveContainer" containerID="5250106705e37da31151211561a16dedf7e81d6814d01f0f08771cbf36725273" Oct 29 22:30:44.319297 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.319268 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7_odh-ai-gateway-infra(56e3163c-2722-49cb-a17d-11af94a932b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" Oct 29 22:30:44.321852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.321063 4952 generic.go:356] "Generic (PLEG): container finished" podID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerID="fcc03dfac458137483b9d19f6ece49a9e9dfc07d32487c5bd1cbe93dcb19e39f" exitCode=60 Oct 29 22:30:44.321852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.321164 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" event={"ID":"19785bf3-8a8c-4346-9a06-01123ccf2378","Type":"ContainerDied","Data":"fcc03dfac458137483b9d19f6ece49a9e9dfc07d32487c5bd1cbe93dcb19e39f"} Oct 29 22:30:44.321852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.321530 4952 scope.go:122] "RemoveContainer" containerID="fcc03dfac458137483b9d19f6ece49a9e9dfc07d32487c5bd1cbe93dcb19e39f" Oct 29 22:30:44.321852 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.321804 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx_odh-ai-gateway-infra(19785bf3-8a8c-4346-9a06-01123ccf2378)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" Oct 29 22:30:44.326093 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.326002 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" event={"ID":"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6","Type":"ContainerStarted","Data":"0c556d2c669ce15890520b6a1c72353b97253b69750e1ca1bea6a69a4717f218"} Oct 29 22:30:44.334421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.334387 4952 generic.go:356] "Generic (PLEG): container finished" podID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerID="d02b270fa0d9aa2f16e67342c358fb48b4ae5a2604088f6c75371a114616a97e" exitCode=60 Oct 29 22:30:44.334553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.334523 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" event={"ID":"dafdb0e0-ba9f-4e05-ad91-1482995b0466","Type":"ContainerDied","Data":"d02b270fa0d9aa2f16e67342c358fb48b4ae5a2604088f6c75371a114616a97e"} Oct 29 22:30:44.334803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.334787 4952 scope.go:122] "RemoveContainer" containerID="d02b270fa0d9aa2f16e67342c358fb48b4ae5a2604088f6c75371a114616a97e" Oct 29 22:30:44.335142 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.335113 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs_odh-ai-gateway-infra(dafdb0e0-ba9f-4e05-ad91-1482995b0466)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" Oct 29 22:30:44.338202 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.338178 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" event={"ID":"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c","Type":"ContainerStarted","Data":"df5f8a94637863b2a3040fbe3fb330ef19b6aa89fa841e185e62cc6f8f0ed18c"} Oct 29 22:30:44.340349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.340322 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" event={"ID":"49f4055f-a3b2-4146-a50a-888fb5d73a35","Type":"ContainerStarted","Data":"8534e218412f20c044e73de209826b0b5ad2d19d24f48c7ce010ba972c742403"} Oct 29 22:30:44.342677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.342657 4952 generic.go:356] "Generic (PLEG): container finished" podID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerID="3a7d606c9c71b67f658358e4027c0896967b18bf7de45ea0e70bed15dbe95029" exitCode=60 Oct 29 22:30:44.342768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.342716 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" event={"ID":"c0e57d7c-7064-43f7-bec3-b92ae3aba56e","Type":"ContainerDied","Data":"3a7d606c9c71b67f658358e4027c0896967b18bf7de45ea0e70bed15dbe95029"} Oct 29 22:30:44.342991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.342975 4952 scope.go:122] "RemoveContainer" containerID="3a7d606c9c71b67f658358e4027c0896967b18bf7de45ea0e70bed15dbe95029" Oct 29 22:30:44.343233 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.343210 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh_odh-ai-gateway-infra(c0e57d7c-7064-43f7-bec3-b92ae3aba56e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" Oct 29 22:30:44.344775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.344739 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" event={"ID":"4c674559-f3d7-47cf-a264-aa41039dc61f","Type":"ContainerStarted","Data":"12024f649c698ac098cae652cafa62a953dd18e8efe66ddcbf2419cb5706afc8"} Oct 29 22:30:44.346752 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.346736 4952 scope.go:122] "RemoveContainer" containerID="34aca47fc2d12fc113a80dfe3762355dc4f025c61d4828658396366d8773854f" Oct 29 22:30:44.347051 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.347027 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2_odh-ai-gateway-infra(95df7b3a-e784-48ec-be58-f5620476c8f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" Oct 29 22:30:44.348317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.348301 4952 scope.go:122] "RemoveContainer" containerID="92645b29c9f68c6b37ddeeb06d7f86d5a08d7234c3ba680682eaf05c6754fdac" Oct 29 22:30:44.348548 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.348524 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd_odh-ai-gateway-infra(e852730b-443d-4644-9a27-de432696d5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" podUID="e852730b-443d-4644-9a27-de432696d5b9" Oct 29 22:30:44.350041 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.350021 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" event={"ID":"575c1d37-7dc0-4270-b023-192457eadac8","Type":"ContainerStarted","Data":"89e0790d52ed2e86b5723c4b7695fa1410f1eb11e823f77c6520c7b30bbb8347"} Oct 29 22:30:44.352059 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.352038 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" event={"ID":"8dc0b51a-0307-4c90-b5ca-cd00ea927a16","Type":"ContainerStarted","Data":"1c500a9f47179182dade8ec3705a008666158e9a58d7b1dbcbefe61afea93abe"} Oct 29 22:30:44.352287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.352270 4952 scope.go:122] "RemoveContainer" containerID="1c500a9f47179182dade8ec3705a008666158e9a58d7b1dbcbefe61afea93abe" Oct 29 22:30:44.352516 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.352497 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2_odh-ai-gateway-infra(8dc0b51a-0307-4c90-b5ca-cd00ea927a16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" Oct 29 22:30:44.353882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.353863 4952 generic.go:356] "Generic (PLEG): container finished" podID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerID="0a5402ab405d8828d08a5349d318a983897893fe0962af9b354427a80cbe1bef" exitCode=60 Oct 29 22:30:44.354040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.353935 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" event={"ID":"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a","Type":"ContainerDied","Data":"0a5402ab405d8828d08a5349d318a983897893fe0962af9b354427a80cbe1bef"} Oct 29 22:30:44.354170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.354154 4952 scope.go:122] "RemoveContainer" containerID="0a5402ab405d8828d08a5349d318a983897893fe0962af9b354427a80cbe1bef" Oct 29 22:30:44.354391 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.354370 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b_odh-ai-gateway-infra(6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" Oct 29 22:30:44.355942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.355921 4952 generic.go:356] "Generic (PLEG): container finished" podID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerID="75b1e2ca0441bbdd1b25b11498c268abd675dd6111ea4a600ba4e0285963b58f" exitCode=60 Oct 29 22:30:44.356030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.355981 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" event={"ID":"f9d92b54-ecdc-4dc5-842b-be8811d0a761","Type":"ContainerDied","Data":"75b1e2ca0441bbdd1b25b11498c268abd675dd6111ea4a600ba4e0285963b58f"} Oct 29 22:30:44.356190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.356174 4952 scope.go:122] "RemoveContainer" containerID="75b1e2ca0441bbdd1b25b11498c268abd675dd6111ea4a600ba4e0285963b58f" Oct 29 22:30:44.356392 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:44.356374 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f_odh-ai-gateway-infra(f9d92b54-ecdc-4dc5-842b-be8811d0a761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" Oct 29 22:30:44.491341 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.491288 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" podStartSLOduration=44.4912736 podStartE2EDuration="44.4912736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:44.486374587 +0000 UTC m=+2909.960963006" watchObservedRunningTime="2026-10-29 22:30:44.4912736 +0000 UTC m=+2909.965862028" Oct 29 22:30:44.591471 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.591447 4952 scope.go:122] "RemoveContainer" containerID="df71961e122b97959fc07f8eefe3089ff6da0cd13b31f827376865ed93298422" Oct 29 22:30:44.607558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.607529 4952 scope.go:122] "RemoveContainer" containerID="a6fac33351ef4b2dd5a65e8e98c258b69ab8af5c12b56987ff35819949b7312a" Oct 29 22:30:44.629628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.629599 4952 scope.go:122] "RemoveContainer" containerID="d477e716f1d86d3822eaf07c1f099c61d927dab930fb05dd4722b5843d809b68" Oct 29 22:30:44.700041 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.699658 4952 scope.go:122] "RemoveContainer" containerID="f16df72a14d2f0f155ef9e2e17dbde6d5dbaba8c27f546871da43059ec97c690" Oct 29 22:30:44.725085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.724993 4952 scope.go:122] "RemoveContainer" containerID="ba277de12abb6626449876fff9940cd20ba130887866a439d01a78d13ce496dd" Oct 29 22:30:44.774464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.774438 4952 scope.go:122] "RemoveContainer" containerID="571f88b95726b89f7097faec7acd14fd51b1c3569a827731002e7b4b81e04763" Oct 29 22:30:44.918443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.918414 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 22:30:44.995292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995201 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d9b425-0133-4d70-a947-7926b68bc0e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d9b425-0133-4d70-a947-7926b68bc0e1-istiod-ca-cert\") pod \"a4d9b425-0133-4d70-a947-7926b68bc0e1\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " Oct 29 22:30:44.995292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995251 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-socket\") pod \"a4d9b425-0133-4d70-a947-7926b68bc0e1\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " Oct 29 22:30:44.995292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995291 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-certs\") pod \"a4d9b425-0133-4d70-a947-7926b68bc0e1\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " Oct 29 22:30:44.995592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995314 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-data\") pod \"a4d9b425-0133-4d70-a947-7926b68bc0e1\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " Oct 29 22:30:44.995592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995375 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-token\" (UniqueName: \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-token\") pod \"a4d9b425-0133-4d70-a947-7926b68bc0e1\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " Oct 29 22:30:44.995592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995495 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-credential-socket\") pod \"a4d9b425-0133-4d70-a947-7926b68bc0e1\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " Oct 29 22:30:44.995770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995637 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-socket" pod "a4d9b425-0133-4d70-a947-7926b68bc0e1" (UID: "a4d9b425-0133-4d70-a947-7926b68bc0e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:44.995832 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995783 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-kube-api-access-blqm6\" (UniqueName: \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-kube-api-access-blqm6\") pod \"a4d9b425-0133-4d70-a947-7926b68bc0e1\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " Oct 29 22:30:44.995893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995853 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-data" pod "a4d9b425-0133-4d70-a947-7926b68bc0e1" (UID: "a4d9b425-0133-4d70-a947-7926b68bc0e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:44.995893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995866 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-certs" pod "a4d9b425-0133-4d70-a947-7926b68bc0e1" (UID: "a4d9b425-0133-4d70-a947-7926b68bc0e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:44.995893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995744 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d9b425-0133-4d70-a947-7926b68bc0e1-istiod-ca-cert" pod "a4d9b425-0133-4d70-a947-7926b68bc0e1" (UID: "a4d9b425-0133-4d70-a947-7926b68bc0e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:44.996088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995881 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-envoy\") pod \"a4d9b425-0133-4d70-a947-7926b68bc0e1\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " Oct 29 22:30:44.996088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995995 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-podinfo\") pod \"a4d9b425-0133-4d70-a947-7926b68bc0e1\" (UID: \"a4d9b425-0133-4d70-a947-7926b68bc0e1\") " Oct 29 22:30:44.996207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.995881 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-credential-socket" pod "a4d9b425-0133-4d70-a947-7926b68bc0e1" (UID: "a4d9b425-0133-4d70-a947-7926b68bc0e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:44.997076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.997047 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d9b425-0133-4d70-a947-7926b68bc0e1-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:44.997198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.997082 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:44.997198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.997101 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:44.997198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.997118 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:44.997198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.997134 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:44.998276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.998250 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-kube-api-access-blqm6" pod "a4d9b425-0133-4d70-a947-7926b68bc0e1" (UID: "a4d9b425-0133-4d70-a947-7926b68bc0e1"). InnerVolumeSpecName "kube-api-access-blqm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:44.998387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.998363 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-podinfo" pod "a4d9b425-0133-4d70-a947-7926b68bc0e1" (UID: "a4d9b425-0133-4d70-a947-7926b68bc0e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:30:44.998454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.998409 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-token" pod "a4d9b425-0133-4d70-a947-7926b68bc0e1" (UID: "a4d9b425-0133-4d70-a947-7926b68bc0e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:44.999044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:44.999025 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-envoy" pod "a4d9b425-0133-4d70-a947-7926b68bc0e1" (UID: "a4d9b425-0133-4d70-a947-7926b68bc0e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:30:45.098821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.098782 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blqm6\" (UniqueName: \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-kube-api-access-blqm6\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.098821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.098819 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.099050 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.098834 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.099050 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.098848 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d9b425-0133-4d70-a947-7926b68bc0e1-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:45.371990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.371950 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" event={"ID":"a4d9b425-0133-4d70-a947-7926b68bc0e1","Type":"ContainerDied","Data":"dd8a679a4d5fd7f20a971fcda0557b10627ca919fed797e5a30658a35fab614e"} Oct 29 22:30:45.372766 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.372024 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv" Oct 29 22:30:45.372766 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.372503 4952 scope.go:122] "RemoveContainer" containerID="79809d795fc4dbe78db8e808d66b7fe263ca0f096d8d349a26b8006598b8a858" Oct 29 22:30:45.375749 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.375422 4952 scope.go:122] "RemoveContainer" containerID="bb504afd4934409689149ffcea778a4ea69d427277c0854a21b47469d248c844" Oct 29 22:30:45.375749 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.375730 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp_odh-ai-gateway-infra(2412c3a1-19b8-4bba-8399-590859b54a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" Oct 29 22:30:45.379493 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.379468 4952 scope.go:122] "RemoveContainer" containerID="f2fa6c40080e8f17796ebeb809674746cf234c52fec2307df9891102335fc391" Oct 29 22:30:45.379865 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.379837 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6_odh-ai-gateway-infra(ae173ee2-aa93-41b3-a205-7bf63dfdba83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" Oct 29 22:30:45.382986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.382965 4952 scope.go:122] "RemoveContainer" containerID="fcc03dfac458137483b9d19f6ece49a9e9dfc07d32487c5bd1cbe93dcb19e39f" Oct 29 22:30:45.383333 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.383305 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx_odh-ai-gateway-infra(19785bf3-8a8c-4346-9a06-01123ccf2378)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" Oct 29 22:30:45.385568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.385536 4952 generic.go:356] "Generic (PLEG): container finished" podID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerID="0c556d2c669ce15890520b6a1c72353b97253b69750e1ca1bea6a69a4717f218" exitCode=60 Oct 29 22:30:45.385677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.385629 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" event={"ID":"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6","Type":"ContainerDied","Data":"0c556d2c669ce15890520b6a1c72353b97253b69750e1ca1bea6a69a4717f218"} Oct 29 22:30:45.386075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.386054 4952 scope.go:122] "RemoveContainer" containerID="0c556d2c669ce15890520b6a1c72353b97253b69750e1ca1bea6a69a4717f218" Oct 29 22:30:45.386409 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.386385 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd_odh-ai-gateway-infra(0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" Oct 29 22:30:45.388875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.388855 4952 scope.go:122] "RemoveContainer" containerID="d02b270fa0d9aa2f16e67342c358fb48b4ae5a2604088f6c75371a114616a97e" Oct 29 22:30:45.389152 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.389130 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs_odh-ai-gateway-infra(dafdb0e0-ba9f-4e05-ad91-1482995b0466)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" Oct 29 22:30:45.402735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.402680 4952 scope.go:122] "RemoveContainer" containerID="90628b1d57f576e02fa0409e403a4eeb5ee446cdbbef182406e7fc691a242357" Oct 29 22:30:45.409350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.409311 4952 generic.go:356] "Generic (PLEG): container finished" podID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerID="df5f8a94637863b2a3040fbe3fb330ef19b6aa89fa841e185e62cc6f8f0ed18c" exitCode=60 Oct 29 22:30:45.409528 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.409430 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" event={"ID":"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c","Type":"ContainerDied","Data":"df5f8a94637863b2a3040fbe3fb330ef19b6aa89fa841e185e62cc6f8f0ed18c"} Oct 29 22:30:45.410171 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.410148 4952 scope.go:122] "RemoveContainer" containerID="df5f8a94637863b2a3040fbe3fb330ef19b6aa89fa841e185e62cc6f8f0ed18c" Oct 29 22:30:45.410997 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.410888 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5_odh-ai-gateway-infra(20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" Oct 29 22:30:45.414171 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.414147 4952 generic.go:356] "Generic (PLEG): container finished" podID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerID="8534e218412f20c044e73de209826b0b5ad2d19d24f48c7ce010ba972c742403" exitCode=60 Oct 29 22:30:45.414271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.414214 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" event={"ID":"49f4055f-a3b2-4146-a50a-888fb5d73a35","Type":"ContainerDied","Data":"8534e218412f20c044e73de209826b0b5ad2d19d24f48c7ce010ba972c742403"} Oct 29 22:30:45.415029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.415012 4952 scope.go:122] "RemoveContainer" containerID="8534e218412f20c044e73de209826b0b5ad2d19d24f48c7ce010ba972c742403" Oct 29 22:30:45.415349 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.415324 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9_odh-ai-gateway-infra(49f4055f-a3b2-4146-a50a-888fb5d73a35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" Oct 29 22:30:45.417629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.417610 4952 scope.go:122] "RemoveContainer" containerID="3a7d606c9c71b67f658358e4027c0896967b18bf7de45ea0e70bed15dbe95029" Oct 29 22:30:45.418000 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.417977 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh_odh-ai-gateway-infra(c0e57d7c-7064-43f7-bec3-b92ae3aba56e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" Oct 29 22:30:45.419717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.419673 4952 generic.go:356] "Generic (PLEG): container finished" podID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerID="12024f649c698ac098cae652cafa62a953dd18e8efe66ddcbf2419cb5706afc8" exitCode=60 Oct 29 22:30:45.419862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.419830 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" event={"ID":"4c674559-f3d7-47cf-a264-aa41039dc61f","Type":"ContainerDied","Data":"12024f649c698ac098cae652cafa62a953dd18e8efe66ddcbf2419cb5706afc8"} Oct 29 22:30:45.420013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.419994 4952 scope.go:122] "RemoveContainer" containerID="12024f649c698ac098cae652cafa62a953dd18e8efe66ddcbf2419cb5706afc8" Oct 29 22:30:45.420460 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.420298 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888550-hmrdh_odh-ai-gateway-infra(4c674559-f3d7-47cf-a264-aa41039dc61f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" Oct 29 22:30:45.428349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.428324 4952 generic.go:356] "Generic (PLEG): container finished" podID="575c1d37-7dc0-4270-b023-192457eadac8" containerID="89e0790d52ed2e86b5723c4b7695fa1410f1eb11e823f77c6520c7b30bbb8347" exitCode=60 Oct 29 22:30:45.428862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.428367 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" event={"ID":"575c1d37-7dc0-4270-b023-192457eadac8","Type":"ContainerDied","Data":"89e0790d52ed2e86b5723c4b7695fa1410f1eb11e823f77c6520c7b30bbb8347"} Oct 29 22:30:45.428862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.428838 4952 scope.go:122] "RemoveContainer" containerID="89e0790d52ed2e86b5723c4b7695fa1410f1eb11e823f77c6520c7b30bbb8347" Oct 29 22:30:45.429344 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.429269 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6_odh-ai-gateway-infra(575c1d37-7dc0-4270-b023-192457eadac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" podUID="575c1d37-7dc0-4270-b023-192457eadac8" Oct 29 22:30:45.446528 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.446490 4952 generic.go:356] "Generic (PLEG): container finished" podID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerID="1c500a9f47179182dade8ec3705a008666158e9a58d7b1dbcbefe61afea93abe" exitCode=60 Oct 29 22:30:45.447280 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.447249 4952 scope.go:122] "RemoveContainer" containerID="1c500a9f47179182dade8ec3705a008666158e9a58d7b1dbcbefe61afea93abe" Oct 29 22:30:45.447796 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.447768 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2_odh-ai-gateway-infra(8dc0b51a-0307-4c90-b5ca-cd00ea927a16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" Oct 29 22:30:45.448546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.448524 4952 scope.go:122] "RemoveContainer" containerID="dd3f51905986ae828d964fe0d6c209531d7aca5ed4995bd0f76fa0b6b585f20d" Oct 29 22:30:45.449072 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.448974 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" event={"ID":"8dc0b51a-0307-4c90-b5ca-cd00ea927a16","Type":"ContainerDied","Data":"1c500a9f47179182dade8ec3705a008666158e9a58d7b1dbcbefe61afea93abe"} Oct 29 22:30:45.449491 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.449469 4952 scope.go:122] "RemoveContainer" containerID="0a5402ab405d8828d08a5349d318a983897893fe0962af9b354427a80cbe1bef" Oct 29 22:30:45.449890 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.449777 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b_odh-ai-gateway-infra(6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" Oct 29 22:30:45.452662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.452417 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv"] Oct 29 22:30:45.454042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.454020 4952 scope.go:122] "RemoveContainer" containerID="75b1e2ca0441bbdd1b25b11498c268abd675dd6111ea4a600ba4e0285963b58f" Oct 29 22:30:45.454361 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.454331 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f_odh-ai-gateway-infra(f9d92b54-ecdc-4dc5-842b-be8811d0a761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" Oct 29 22:30:45.459471 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.459444 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-6r7xv"] Oct 29 22:30:45.474316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.474290 4952 scope.go:122] "RemoveContainer" containerID="dcae8b178c7e3c2e7a4d3435532b3d4390ed15d26363fd89f004a0eca4ae7e91" Oct 29 22:30:45.492600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.492578 4952 scope.go:122] "RemoveContainer" containerID="4d9a22e83057871b137f689af259beb519544d2f046b251726bef737d3a8bd3f" Oct 29 22:30:45.508081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.508059 4952 scope.go:122] "RemoveContainer" containerID="5b40a2680558a445a69347c2a20a62602415139b15495ce1a5db42748ab54a1d" Oct 29 22:30:45.535489 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:45.535100 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:30:45.535489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:45.535133 4952 scope.go:122] "RemoveContainer" containerID="e4bec9c747242a08a4784f3499d9fece9ec7cbceed50f1c0cdd7b2d2ca253b4c" Oct 29 22:30:46.465980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:46.465949 4952 scope.go:122] "RemoveContainer" containerID="0c556d2c669ce15890520b6a1c72353b97253b69750e1ca1bea6a69a4717f218" Oct 29 22:30:46.466966 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:46.466933 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd_odh-ai-gateway-infra(0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" Oct 29 22:30:46.468620 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:46.468590 4952 scope.go:122] "RemoveContainer" containerID="df5f8a94637863b2a3040fbe3fb330ef19b6aa89fa841e185e62cc6f8f0ed18c" Oct 29 22:30:46.469031 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:46.469006 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5_odh-ai-gateway-infra(20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" Oct 29 22:30:46.470877 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:46.470857 4952 scope.go:122] "RemoveContainer" containerID="8534e218412f20c044e73de209826b0b5ad2d19d24f48c7ce010ba972c742403" Oct 29 22:30:46.471227 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:46.471203 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9_odh-ai-gateway-infra(49f4055f-a3b2-4146-a50a-888fb5d73a35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" Oct 29 22:30:46.472821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:46.472800 4952 scope.go:122] "RemoveContainer" containerID="12024f649c698ac098cae652cafa62a953dd18e8efe66ddcbf2419cb5706afc8" Oct 29 22:30:46.473192 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:46.473164 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888550-hmrdh_odh-ai-gateway-infra(4c674559-f3d7-47cf-a264-aa41039dc61f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" Oct 29 22:30:46.474721 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:46.474637 4952 scope.go:122] "RemoveContainer" containerID="89e0790d52ed2e86b5723c4b7695fa1410f1eb11e823f77c6520c7b30bbb8347" Oct 29 22:30:46.475006 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:46.474980 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6_odh-ai-gateway-infra(575c1d37-7dc0-4270-b023-192457eadac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" podUID="575c1d37-7dc0-4270-b023-192457eadac8" Oct 29 22:30:46.476688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:46.476658 4952 scope.go:122] "RemoveContainer" containerID="1c500a9f47179182dade8ec3705a008666158e9a58d7b1dbcbefe61afea93abe" Oct 29 22:30:46.476990 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:46.476958 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2_odh-ai-gateway-infra(8dc0b51a-0307-4c90-b5ca-cd00ea927a16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" Oct 29 22:30:46.540422 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:46.540386 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:46.668482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:46.668443 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" path="/var/lib/kubelet/pods/a4d9b425-0133-4d70-a947-7926b68bc0e1/volumes" Oct 29 22:30:46.794529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:46.794418 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:48.121510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:48.118875 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:49.789930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:49.787173 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:50.154926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:50.154747 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:50.594676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:50.594626 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:50.988519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:50.988437 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g"] Oct 29 22:30:50.989945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:50.989917 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" Oct 29 22:30:50.990035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:50.989947 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" Oct 29 22:30:50.990035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:50.989986 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" Oct 29 22:30:50.990035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:50.989993 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" Oct 29 22:30:50.990248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:50.990229 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" containerName="istio-proxy" Oct 29 22:30:50.990313 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:50.990259 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d9b425-0133-4d70-a947-7926b68bc0e1" containerName="istio-proxy" Oct 29 22:30:50.996760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:50.996734 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.021058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.021024 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g"] Oct 29 22:30:51.029162 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.029134 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-90f46c-openshift-default-dockercfg-2gcfd\"" Oct 29 22:30:51.105653 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.105616 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:51.111615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.111590 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-certs\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.111735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.111626 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-podinfo\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.111890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.111860 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-credential-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.112029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.112009 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istiod-ca-cert\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.112079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.112060 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.112178 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.112156 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-token\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.112228 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.112212 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-envoy\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.112335 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.112318 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9fc\" (UniqueName: \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-kube-api-access-zm9fc\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.112417 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.112404 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-data\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.220934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.219983 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9fc\" (UniqueName: \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-kube-api-access-zm9fc\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.220934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.220083 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-data\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.220934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.220176 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-certs\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.220934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.220221 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-podinfo\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.220934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.220384 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-credential-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.242982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.220490 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istiod-ca-cert\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.242982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.228083 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.242982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.228160 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-token\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.242982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.228228 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-envoy\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.243776 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.243741 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-certs\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.277930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.260646 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-data\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.277930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.261530 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istiod-ca-cert\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.277930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.261855 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-credential-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.307930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.278811 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-envoy\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.309008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.286591 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-podinfo\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.309173 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.303534 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9fc\" (UniqueName: \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-kube-api-access-zm9fc\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.310922 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.310877 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-socket\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.339927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.332518 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-token\") pod \"e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.629188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.629154 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:51.844929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:51.843929 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:52.084451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:52.084246 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:52.111355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:52.110592 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g"] Oct 29 22:30:52.135179 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:52.119785 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:30:52.135179 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:52.129058 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:30:52.135179 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:52.129156 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:30:52.564796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:52.564755 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" event={"ID":"d84b5278-d243-470c-abf8-e6e8e9aa3e62","Type":"ContainerStarted","Data":"52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21"} Oct 29 22:30:52.565101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:52.565068 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" event={"ID":"d84b5278-d243-470c-abf8-e6e8e9aa3e62","Type":"ContainerStarted","Data":"51f100f67753f71b4f44532487092d2353f74b75019df34d6531450bd108613a"} Oct 29 22:30:52.621005 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:52.620939 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" podStartSLOduration=2.620895367 podStartE2EDuration="2.620895367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:52.613234669 +0000 UTC m=+2918.087823112" watchObservedRunningTime="2026-10-29 22:30:52.620895367 +0000 UTC m=+2918.095483792" Oct 29 22:30:52.631120 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:52.631084 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:53.638241 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:53.638203 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:53.910375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:53.910281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:54.583559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:54.583525 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:54.584786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:54.584763 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:30:54.637458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:54.637416 4952 scope.go:122] "RemoveContainer" containerID="d02b270fa0d9aa2f16e67342c358fb48b4ae5a2604088f6c75371a114616a97e" Oct 29 22:30:54.637684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:54.637471 4952 scope.go:122] "RemoveContainer" containerID="8534e218412f20c044e73de209826b0b5ad2d19d24f48c7ce010ba972c742403" Oct 29 22:30:54.637684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:54.637673 4952 scope.go:122] "RemoveContainer" containerID="92645b29c9f68c6b37ddeeb06d7f86d5a08d7234c3ba680682eaf05c6754fdac" Oct 29 22:30:54.638930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:54.638257 4952 scope.go:122] "RemoveContainer" containerID="0a5402ab405d8828d08a5349d318a983897893fe0962af9b354427a80cbe1bef" Oct 29 22:30:55.118041 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.117990 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:55.369173 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.369140 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:55.648927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.637808 4952 scope.go:122] "RemoveContainer" containerID="0c556d2c669ce15890520b6a1c72353b97253b69750e1ca1bea6a69a4717f218" Oct 29 22:30:55.648927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.638377 4952 scope.go:122] "RemoveContainer" containerID="3a7d606c9c71b67f658358e4027c0896967b18bf7de45ea0e70bed15dbe95029" Oct 29 22:30:55.648927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.639385 4952 scope.go:122] "RemoveContainer" containerID="5250106705e37da31151211561a16dedf7e81d6814d01f0f08771cbf36725273" Oct 29 22:30:55.648927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.640195 4952 scope.go:122] "RemoveContainer" containerID="1c500a9f47179182dade8ec3705a008666158e9a58d7b1dbcbefe61afea93abe" Oct 29 22:30:55.680932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.679120 4952 scope.go:122] "RemoveContainer" containerID="2f1884cfb0fb3388c0652d56e1c31b9d078825c6cac7c8baccde21b4a86ba805" Oct 29 22:30:55.680932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.679960 4952 scope.go:122] "RemoveContainer" containerID="f2fa6c40080e8f17796ebeb809674746cf234c52fec2307df9891102335fc391" Oct 29 22:30:55.712558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.699719 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" event={"ID":"dafdb0e0-ba9f-4e05-ad91-1482995b0466","Type":"ContainerStarted","Data":"2f630e71e3e70587900998384f745baaf8be3cf6b6f67f1bf86ed5826791df51"} Oct 29 22:30:55.846639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.842031 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" event={"ID":"49f4055f-a3b2-4146-a50a-888fb5d73a35","Type":"ContainerStarted","Data":"b0d5ff5ec41f7eb5e79d407dfd2ce0c0bb46bc132d2b55d766c10dd725f6105f"} Oct 29 22:30:55.872104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.872062 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" event={"ID":"e852730b-443d-4644-9a27-de432696d5b9","Type":"ContainerStarted","Data":"08c2b108ddcd5a6005ddc18a71ab60b1e34be460cdb8ca7d2866f42611ecc953"} Oct 29 22:30:55.907146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:55.907108 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" event={"ID":"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a","Type":"ContainerStarted","Data":"fc8cfb037f235c2b7d8f578f20a998fc9232211e7d433dd7edc04998f51c9c8f"} Oct 29 22:30:56.109515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.109437 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:56.638018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.637985 4952 scope.go:122] "RemoveContainer" containerID="34aca47fc2d12fc113a80dfe3762355dc4f025c61d4828658396366d8773854f" Oct 29 22:30:56.639859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.639829 4952 scope.go:122] "RemoveContainer" containerID="75b1e2ca0441bbdd1b25b11498c268abd675dd6111ea4a600ba4e0285963b58f" Oct 29 22:30:56.813315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.813276 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs"] Oct 29 22:30:56.908345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.908303 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9"] Oct 29 22:30:56.922535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.922481 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" event={"ID":"56e3163c-2722-49cb-a17d-11af94a932b5","Type":"ContainerStarted","Data":"9bd7ae203fc6162aeca5df63e8b5bc635a6f0f696dd399ff42b2848f69fb6656"} Oct 29 22:30:56.930890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.930522 4952 generic.go:356] "Generic (PLEG): container finished" podID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerID="2f630e71e3e70587900998384f745baaf8be3cf6b6f67f1bf86ed5826791df51" exitCode=60 Oct 29 22:30:56.930890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.930634 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" event={"ID":"dafdb0e0-ba9f-4e05-ad91-1482995b0466","Type":"ContainerDied","Data":"2f630e71e3e70587900998384f745baaf8be3cf6b6f67f1bf86ed5826791df51"} Oct 29 22:30:56.930890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.930694 4952 scope.go:122] "RemoveContainer" containerID="d02b270fa0d9aa2f16e67342c358fb48b4ae5a2604088f6c75371a114616a97e" Oct 29 22:30:56.951004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.950973 4952 generic.go:356] "Generic (PLEG): container finished" podID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerID="b0d5ff5ec41f7eb5e79d407dfd2ce0c0bb46bc132d2b55d766c10dd725f6105f" exitCode=60 Oct 29 22:30:56.951244 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.951224 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" event={"ID":"49f4055f-a3b2-4146-a50a-888fb5d73a35","Type":"ContainerDied","Data":"b0d5ff5ec41f7eb5e79d407dfd2ce0c0bb46bc132d2b55d766c10dd725f6105f"} Oct 29 22:30:56.957981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.957951 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" event={"ID":"c0e57d7c-7064-43f7-bec3-b92ae3aba56e","Type":"ContainerStarted","Data":"e9be1ffd81d2b088283f007d9a852814dfce1b69c9a2e0ad84a6383c5279ff75"} Oct 29 22:30:56.964370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.964341 4952 generic.go:356] "Generic (PLEG): container finished" podID="e852730b-443d-4644-9a27-de432696d5b9" containerID="08c2b108ddcd5a6005ddc18a71ab60b1e34be460cdb8ca7d2866f42611ecc953" exitCode=60 Oct 29 22:30:56.964595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.964577 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" event={"ID":"e852730b-443d-4644-9a27-de432696d5b9","Type":"ContainerDied","Data":"08c2b108ddcd5a6005ddc18a71ab60b1e34be460cdb8ca7d2866f42611ecc953"} Oct 29 22:30:56.972199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.970881 4952 generic.go:356] "Generic (PLEG): container finished" podID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerID="fc8cfb037f235c2b7d8f578f20a998fc9232211e7d433dd7edc04998f51c9c8f" exitCode=60 Oct 29 22:30:56.972199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.971827 4952 scope.go:122] "RemoveContainer" containerID="08c2b108ddcd5a6005ddc18a71ab60b1e34be460cdb8ca7d2866f42611ecc953" Oct 29 22:30:56.972199 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:56.972137 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd_odh-ai-gateway-infra(e852730b-443d-4644-9a27-de432696d5b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" podUID="e852730b-443d-4644-9a27-de432696d5b9" Oct 29 22:30:56.972724 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.972678 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" event={"ID":"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a","Type":"ContainerDied","Data":"fc8cfb037f235c2b7d8f578f20a998fc9232211e7d433dd7edc04998f51c9c8f"} Oct 29 22:30:56.973026 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.973005 4952 scope.go:122] "RemoveContainer" containerID="fc8cfb037f235c2b7d8f578f20a998fc9232211e7d433dd7edc04998f51c9c8f" Oct 29 22:30:56.973340 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:56.973221 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b_odh-ai-gateway-infra(6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" Oct 29 22:30:56.976079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.976056 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd"] Oct 29 22:30:56.989935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:56.989828 4952 scope.go:122] "RemoveContainer" containerID="8534e218412f20c044e73de209826b0b5ad2d19d24f48c7ce010ba972c742403" Oct 29 22:30:57.002107 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.002077 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b"] Oct 29 22:30:57.011717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.011679 4952 scope.go:122] "RemoveContainer" containerID="92645b29c9f68c6b37ddeeb06d7f86d5a08d7234c3ba680682eaf05c6754fdac" Oct 29 22:30:57.080188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.080134 4952 scope.go:122] "RemoveContainer" containerID="0a5402ab405d8828d08a5349d318a983897893fe0962af9b354427a80cbe1bef" Oct 29 22:30:57.264321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.264220 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" Oct 29 22:30:57.285379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.285343 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafdb0e0-ba9f-4e05-ad91-1482995b0466-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dafdb0e0-ba9f-4e05-ad91-1482995b0466-maas-api-service-ca\") pod \"dafdb0e0-ba9f-4e05-ad91-1482995b0466\" (UID: \"dafdb0e0-ba9f-4e05-ad91-1482995b0466\") " Oct 29 22:30:57.285585 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.285501 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafdb0e0-ba9f-4e05-ad91-1482995b0466-kube-api-access-qdbmq\" (UniqueName: \"kubernetes.io/projected/dafdb0e0-ba9f-4e05-ad91-1482995b0466-kube-api-access-qdbmq\") pod \"dafdb0e0-ba9f-4e05-ad91-1482995b0466\" (UID: \"dafdb0e0-ba9f-4e05-ad91-1482995b0466\") " Oct 29 22:30:57.286082 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.286055 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafdb0e0-ba9f-4e05-ad91-1482995b0466-maas-api-service-ca" pod "dafdb0e0-ba9f-4e05-ad91-1482995b0466" (UID: "dafdb0e0-ba9f-4e05-ad91-1482995b0466"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:57.289521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.289166 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafdb0e0-ba9f-4e05-ad91-1482995b0466-kube-api-access-qdbmq" pod "dafdb0e0-ba9f-4e05-ad91-1482995b0466" (UID: "dafdb0e0-ba9f-4e05-ad91-1482995b0466"). InnerVolumeSpecName "kube-api-access-qdbmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:57.388924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.388885 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdbmq\" (UniqueName: \"kubernetes.io/projected/dafdb0e0-ba9f-4e05-ad91-1482995b0466-kube-api-access-qdbmq\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:57.388924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.388927 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dafdb0e0-ba9f-4e05-ad91-1482995b0466-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:57.389244 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.389230 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" Oct 29 22:30:57.489835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.489734 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f4055f-a3b2-4146-a50a-888fb5d73a35-kube-api-access-6ft6c\" (UniqueName: \"kubernetes.io/projected/49f4055f-a3b2-4146-a50a-888fb5d73a35-kube-api-access-6ft6c\") pod \"49f4055f-a3b2-4146-a50a-888fb5d73a35\" (UID: \"49f4055f-a3b2-4146-a50a-888fb5d73a35\") " Oct 29 22:30:57.490057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.489830 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f4055f-a3b2-4146-a50a-888fb5d73a35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f4055f-a3b2-4146-a50a-888fb5d73a35-maas-api-service-ca\") pod \"49f4055f-a3b2-4146-a50a-888fb5d73a35\" (UID: \"49f4055f-a3b2-4146-a50a-888fb5d73a35\") " Oct 29 22:30:57.491008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.490592 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f4055f-a3b2-4146-a50a-888fb5d73a35-maas-api-service-ca" pod "49f4055f-a3b2-4146-a50a-888fb5d73a35" (UID: "49f4055f-a3b2-4146-a50a-888fb5d73a35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:57.491008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.490974 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f4055f-a3b2-4146-a50a-888fb5d73a35-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:57.493919 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.493882 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f4055f-a3b2-4146-a50a-888fb5d73a35-kube-api-access-6ft6c" pod "49f4055f-a3b2-4146-a50a-888fb5d73a35" (UID: "49f4055f-a3b2-4146-a50a-888fb5d73a35"). InnerVolumeSpecName "kube-api-access-6ft6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:57.593127 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.593086 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ft6c\" (UniqueName: \"kubernetes.io/projected/49f4055f-a3b2-4146-a50a-888fb5d73a35-kube-api-access-6ft6c\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:57.636860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.636832 4952 scope.go:122] "RemoveContainer" containerID="bb504afd4934409689149ffcea778a4ea69d427277c0854a21b47469d248c844" Oct 29 22:30:57.637292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.637251 4952 scope.go:122] "RemoveContainer" containerID="fcc03dfac458137483b9d19f6ece49a9e9dfc07d32487c5bd1cbe93dcb19e39f" Oct 29 22:30:57.639479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.639077 4952 scope.go:122] "RemoveContainer" containerID="c68d7bdfa30fac7bed79500c97e2bcc5075dcaeb190b6ad8923b41e188145359" Oct 29 22:30:57.800493 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.800460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:57.973106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.973068 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:57.994576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.994545 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7"] Oct 29 22:30:57.998585 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.998558 4952 generic.go:356] "Generic (PLEG): container finished" podID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerID="721e4697289f5ff92c2d80908b4d89ffb0803fcdf36ccc5542fbda683ffc5cbf" exitCode=60 Oct 29 22:30:57.998717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.998621 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" event={"ID":"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6","Type":"ContainerDied","Data":"721e4697289f5ff92c2d80908b4d89ffb0803fcdf36ccc5542fbda683ffc5cbf"} Oct 29 22:30:57.998717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.998675 4952 scope.go:122] "RemoveContainer" containerID="0c556d2c669ce15890520b6a1c72353b97253b69750e1ca1bea6a69a4717f218" Oct 29 22:30:57.999269 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:57.999249 4952 scope.go:122] "RemoveContainer" containerID="721e4697289f5ff92c2d80908b4d89ffb0803fcdf36ccc5542fbda683ffc5cbf" Oct 29 22:30:57.999824 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:57.999712 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd_odh-ai-gateway-infra(0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" Oct 29 22:30:58.000645 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.000616 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" event={"ID":"dafdb0e0-ba9f-4e05-ad91-1482995b0466","Type":"ContainerDied","Data":"7a2c1a0730e32d71632ad698e322cd249e546ed1b143955df23d21094a13c9ec"} Oct 29 22:30:58.000775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.000706 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs" Oct 29 22:30:58.007070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.005990 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" event={"ID":"49f4055f-a3b2-4146-a50a-888fb5d73a35","Type":"ContainerDied","Data":"3a1f921209eeb49ab5f3186225c63e60c7adee9d34773828d6b61d63a280da90"} Oct 29 22:30:58.007070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.006098 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9" Oct 29 22:30:58.037674 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.033182 4952 scope.go:122] "RemoveContainer" containerID="2f630e71e3e70587900998384f745baaf8be3cf6b6f67f1bf86ed5826791df51" Oct 29 22:30:58.051782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.042780 4952 generic.go:356] "Generic (PLEG): container finished" podID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerID="e9be1ffd81d2b088283f007d9a852814dfce1b69c9a2e0ad84a6383c5279ff75" exitCode=60 Oct 29 22:30:58.051782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.042885 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" event={"ID":"c0e57d7c-7064-43f7-bec3-b92ae3aba56e","Type":"ContainerDied","Data":"e9be1ffd81d2b088283f007d9a852814dfce1b69c9a2e0ad84a6383c5279ff75"} Oct 29 22:30:58.051782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.043468 4952 scope.go:122] "RemoveContainer" containerID="e9be1ffd81d2b088283f007d9a852814dfce1b69c9a2e0ad84a6383c5279ff75" Oct 29 22:30:58.051782 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:58.043952 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh_odh-ai-gateway-infra(c0e57d7c-7064-43f7-bec3-b92ae3aba56e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" Oct 29 22:30:58.063148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.061060 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" event={"ID":"95df7b3a-e784-48ec-be58-f5620476c8f9","Type":"ContainerStarted","Data":"39a44c9b976951cb897411cad6f73d92ced50415cfe5dce075872236001561ac"} Oct 29 22:30:58.063148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.061685 4952 scope.go:122] "RemoveContainer" containerID="39a44c9b976951cb897411cad6f73d92ced50415cfe5dce075872236001561ac" Oct 29 22:30:58.063458 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:58.063185 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2_odh-ai-gateway-infra(95df7b3a-e784-48ec-be58-f5620476c8f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" Oct 29 22:30:58.346998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.202161 4952 generic.go:356] "Generic (PLEG): container finished" podID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerID="b91ce6025b05007bcf2a8eab36c82a5c476ea135f7ace7729cb7b67e475dc6a4" exitCode=60 Oct 29 22:30:58.346998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.202291 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" event={"ID":"8dc0b51a-0307-4c90-b5ca-cd00ea927a16","Type":"ContainerDied","Data":"b91ce6025b05007bcf2a8eab36c82a5c476ea135f7ace7729cb7b67e475dc6a4"} Oct 29 22:30:58.346998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.202402 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:58.346998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.216987 4952 scope.go:122] "RemoveContainer" containerID="b91ce6025b05007bcf2a8eab36c82a5c476ea135f7ace7729cb7b67e475dc6a4" Oct 29 22:30:58.346998 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:58.217509 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2_odh-ai-gateway-infra(8dc0b51a-0307-4c90-b5ca-cd00ea927a16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" Oct 29 22:30:58.512959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.512110 4952 generic.go:356] "Generic (PLEG): container finished" podID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerID="099488ab71fb43fa5a9b11cd828363973294ef70f918dfdc1773f67538aa9ed3" exitCode=60 Oct 29 22:30:58.512959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.512336 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" event={"ID":"f9d92b54-ecdc-4dc5-842b-be8811d0a761","Type":"ContainerDied","Data":"099488ab71fb43fa5a9b11cd828363973294ef70f918dfdc1773f67538aa9ed3"} Oct 29 22:30:58.512959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.512853 4952 scope.go:122] "RemoveContainer" containerID="099488ab71fb43fa5a9b11cd828363973294ef70f918dfdc1773f67538aa9ed3" Oct 29 22:30:58.515677 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:58.513847 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f_odh-ai-gateway-infra(f9d92b54-ecdc-4dc5-842b-be8811d0a761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" Oct 29 22:30:58.517012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.516987 4952 generic.go:356] "Generic (PLEG): container finished" podID="06ea33cb-020d-415a-b518-37018182741a" containerID="71c16d991ef0ca9a14e1ecf795fcd0b82866b0cc84a39797cdc6b1758d5e56fc" exitCode=60 Oct 29 22:30:58.517249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.517233 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" event={"ID":"06ea33cb-020d-415a-b518-37018182741a","Type":"ContainerDied","Data":"71c16d991ef0ca9a14e1ecf795fcd0b82866b0cc84a39797cdc6b1758d5e56fc"} Oct 29 22:30:58.517819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.517802 4952 scope.go:122] "RemoveContainer" containerID="71c16d991ef0ca9a14e1ecf795fcd0b82866b0cc84a39797cdc6b1758d5e56fc" Oct 29 22:30:58.518390 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:58.518370 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp_odh-ai-gateway-infra(06ea33cb-020d-415a-b518-37018182741a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" podUID="06ea33cb-020d-415a-b518-37018182741a" Oct 29 22:30:58.520834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.520813 4952 generic.go:356] "Generic (PLEG): container finished" podID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerID="48cd103a887360d61e37fa3c83836caa26fbc7dcd042bd99296cf164b151f827" exitCode=60 Oct 29 22:30:58.525075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.521032 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" event={"ID":"ae173ee2-aa93-41b3-a205-7bf63dfdba83","Type":"ContainerDied","Data":"48cd103a887360d61e37fa3c83836caa26fbc7dcd042bd99296cf164b151f827"} Oct 29 22:30:58.529281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.525730 4952 scope.go:122] "RemoveContainer" containerID="48cd103a887360d61e37fa3c83836caa26fbc7dcd042bd99296cf164b151f827" Oct 29 22:30:58.534029 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:58.532718 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6_odh-ai-gateway-infra(ae173ee2-aa93-41b3-a205-7bf63dfdba83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" Oct 29 22:30:58.538220 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.538189 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:58.543258 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.543161 4952 scope.go:122] "RemoveContainer" containerID="b0d5ff5ec41f7eb5e79d407dfd2ce0c0bb46bc132d2b55d766c10dd725f6105f" Oct 29 22:30:58.550754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.550725 4952 generic.go:356] "Generic (PLEG): container finished" podID="56e3163c-2722-49cb-a17d-11af94a932b5" containerID="9bd7ae203fc6162aeca5df63e8b5bc635a6f0f696dd399ff42b2848f69fb6656" exitCode=60 Oct 29 22:30:58.550956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.550936 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" event={"ID":"56e3163c-2722-49cb-a17d-11af94a932b5","Type":"ContainerDied","Data":"9bd7ae203fc6162aeca5df63e8b5bc635a6f0f696dd399ff42b2848f69fb6656"} Oct 29 22:30:58.608618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.608077 4952 scope.go:122] "RemoveContainer" containerID="3a7d606c9c71b67f658358e4027c0896967b18bf7de45ea0e70bed15dbe95029" Oct 29 22:30:58.614582 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.614550 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh"] Oct 29 22:30:58.694532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.683842 4952 scope.go:122] "RemoveContainer" containerID="1c500a9f47179182dade8ec3705a008666158e9a58d7b1dbcbefe61afea93abe" Oct 29 22:30:58.694532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.686387 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs"] Oct 29 22:30:58.694532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.686428 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bef83b-29888550-ftkfs"] Oct 29 22:30:58.694532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.686561 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9"] Oct 29 22:30:58.694532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.687119 4952 scope.go:122] "RemoveContainer" containerID="89e0790d52ed2e86b5723c4b7695fa1410f1eb11e823f77c6520c7b30bbb8347" Oct 29 22:30:58.694532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.688346 4952 scope.go:122] "RemoveContainer" containerID="12024f649c698ac098cae652cafa62a953dd18e8efe66ddcbf2419cb5706afc8" Oct 29 22:30:58.694532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.688388 4952 scope.go:122] "RemoveContainer" containerID="df5f8a94637863b2a3040fbe3fb330ef19b6aa89fa841e185e62cc6f8f0ed18c" Oct 29 22:30:58.694532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.692699 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-1e74c3-29888550-wbzw9"] Oct 29 22:30:58.780049 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.780021 4952 scope.go:122] "RemoveContainer" containerID="75b1e2ca0441bbdd1b25b11498c268abd675dd6111ea4a600ba4e0285963b58f" Oct 29 22:30:58.780728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.780629 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:58.893552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.893522 4952 scope.go:122] "RemoveContainer" containerID="2f1884cfb0fb3388c0652d56e1c31b9d078825c6cac7c8baccde21b4a86ba805" Oct 29 22:30:58.907090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.907071 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" Oct 29 22:30:58.943372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.931497 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" Oct 29 22:30:58.943372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:58.942287 4952 scope.go:122] "RemoveContainer" containerID="f2fa6c40080e8f17796ebeb809674746cf234c52fec2307df9891102335fc391" Oct 29 22:30:59.078516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.078475 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e3163c-2722-49cb-a17d-11af94a932b5-kube-api-access-6rc2b\" (UniqueName: \"kubernetes.io/projected/56e3163c-2722-49cb-a17d-11af94a932b5-kube-api-access-6rc2b\") pod \"56e3163c-2722-49cb-a17d-11af94a932b5\" (UID: \"56e3163c-2722-49cb-a17d-11af94a932b5\") " Oct 29 22:30:59.079002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.078538 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e852730b-443d-4644-9a27-de432696d5b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e852730b-443d-4644-9a27-de432696d5b9-maas-api-service-ca\") pod \"e852730b-443d-4644-9a27-de432696d5b9\" (UID: \"e852730b-443d-4644-9a27-de432696d5b9\") " Oct 29 22:30:59.079002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.078629 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e3163c-2722-49cb-a17d-11af94a932b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e3163c-2722-49cb-a17d-11af94a932b5-maas-api-service-ca\") pod \"56e3163c-2722-49cb-a17d-11af94a932b5\" (UID: \"56e3163c-2722-49cb-a17d-11af94a932b5\") " Oct 29 22:30:59.079002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.078812 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e852730b-443d-4644-9a27-de432696d5b9-kube-api-access-vk8dk\" (UniqueName: \"kubernetes.io/projected/e852730b-443d-4644-9a27-de432696d5b9-kube-api-access-vk8dk\") pod \"e852730b-443d-4644-9a27-de432696d5b9\" (UID: \"e852730b-443d-4644-9a27-de432696d5b9\") " Oct 29 22:30:59.079002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.078976 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd"] Oct 29 22:30:59.079487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.079461 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e852730b-443d-4644-9a27-de432696d5b9-maas-api-service-ca" pod "e852730b-443d-4644-9a27-de432696d5b9" (UID: "e852730b-443d-4644-9a27-de432696d5b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:59.080095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.079811 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e3163c-2722-49cb-a17d-11af94a932b5-maas-api-service-ca" pod "56e3163c-2722-49cb-a17d-11af94a932b5" (UID: "56e3163c-2722-49cb-a17d-11af94a932b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:59.080095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.080049 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e852730b-443d-4644-9a27-de432696d5b9-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:59.080095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.080072 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e3163c-2722-49cb-a17d-11af94a932b5-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:59.082262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.082238 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e3163c-2722-49cb-a17d-11af94a932b5-kube-api-access-6rc2b" pod "56e3163c-2722-49cb-a17d-11af94a932b5" (UID: "56e3163c-2722-49cb-a17d-11af94a932b5"). InnerVolumeSpecName "kube-api-access-6rc2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:59.082426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.082403 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e852730b-443d-4644-9a27-de432696d5b9-kube-api-access-vk8dk" pod "e852730b-443d-4644-9a27-de432696d5b9" (UID: "e852730b-443d-4644-9a27-de432696d5b9"). InnerVolumeSpecName "kube-api-access-vk8dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:59.083476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.083453 4952 scope.go:122] "RemoveContainer" containerID="5250106705e37da31151211561a16dedf7e81d6814d01f0f08771cbf36725273" Oct 29 22:30:59.181800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.181767 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rc2b\" (UniqueName: \"kubernetes.io/projected/56e3163c-2722-49cb-a17d-11af94a932b5-kube-api-access-6rc2b\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:59.181800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.181801 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk8dk\" (UniqueName: \"kubernetes.io/projected/e852730b-443d-4644-9a27-de432696d5b9-kube-api-access-vk8dk\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.195310 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt"] Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196655 4952 cpu_manager.go:414] "containerMap: removing container" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196677 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196693 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e852730b-443d-4644-9a27-de432696d5b9" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196702 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e852730b-443d-4644-9a27-de432696d5b9" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196720 4952 cpu_manager.go:414] "containerMap: removing container" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196728 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196746 4952 cpu_manager.go:414] "containerMap: removing container" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196753 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196785 4952 cpu_manager.go:414] "containerMap: removing container" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196794 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196819 4952 cpu_manager.go:414] "containerMap: removing container" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196826 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196852 4952 cpu_manager.go:414] "containerMap: removing container" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196859 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196868 4952 cpu_manager.go:414] "containerMap: removing container" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196876 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196945 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e852730b-443d-4644-9a27-de432696d5b9" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.196952 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e852730b-443d-4644-9a27-de432696d5b9" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.197167 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.197180 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.197191 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e852730b-443d-4644-9a27-de432696d5b9" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.197203 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.197215 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e852730b-443d-4644-9a27-de432696d5b9" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.197228 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.197239 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.197253 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerName="cleanup" Oct 29 22:30:59.199015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.197263 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" containerName="cleanup" Oct 29 22:30:59.203895 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.203865 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.226063 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.226018 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-26abe8-openshift-default-dockercfg-frqz6\"" Oct 29 22:30:59.229404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.229369 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt"] Oct 29 22:30:59.245915 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.245845 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2"] Oct 29 22:30:59.361682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.360352 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:59.396494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.386287 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-credential-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.396494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.386369 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f904684-fe8c-4e96-befb-229dbf30522f-istio-podinfo\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.396494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.386402 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-certs\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.396494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.386497 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-data\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.396494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.386579 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-envoy\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.396494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.386659 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f904684-fe8c-4e96-befb-229dbf30522f-istiod-ca-cert\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.396494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.386748 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9x7x\" (UniqueName: \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-kube-api-access-p9x7x\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.396494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.386779 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.396494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.386964 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-istio-token\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.439934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.431120 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" Oct 29 22:30:59.513926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.491198 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f904684-fe8c-4e96-befb-229dbf30522f-istiod-ca-cert\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.522928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.514276 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9x7x\" (UniqueName: \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-kube-api-access-p9x7x\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.523252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.523212 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.535930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.523568 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-istio-token\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.536442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.536410 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-credential-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.536622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.536607 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f904684-fe8c-4e96-befb-229dbf30522f-istio-podinfo\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.672351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.668001 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-certs\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.672351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.668142 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-data\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.672351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.668240 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-envoy\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.672351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.562191 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:59.672351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.539656 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-credential-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.672351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.668855 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-certs\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.672351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.498311 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f904684-fe8c-4e96-befb-229dbf30522f-istiod-ca-cert\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.672351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.669369 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-data\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.672351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.524336 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-socket\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.736258 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.736152 4952 generic.go:356] "Generic (PLEG): container finished" podID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerID="1c25be5be81f9dd4afc0a2168eb46c1f519841aeeef785cc88a3eed3fc0035a5" exitCode=60 Oct 29 22:30:59.739951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.736214 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" event={"ID":"d669ddd4-8a30-4078-9c55-870d0f8d948e","Type":"ContainerDied","Data":"1c25be5be81f9dd4afc0a2168eb46c1f519841aeeef785cc88a3eed3fc0035a5"} Oct 29 22:30:59.739951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.736569 4952 scope.go:122] "RemoveContainer" containerID="c68d7bdfa30fac7bed79500c97e2bcc5075dcaeb190b6ad8923b41e188145359" Oct 29 22:30:59.745600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.743187 4952 scope.go:122] "RemoveContainer" containerID="1c25be5be81f9dd4afc0a2168eb46c1f519841aeeef785cc88a3eed3fc0035a5" Oct 29 22:30:59.745600 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:59.743943 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq_odh-ai-gateway-infra(d669ddd4-8a30-4078-9c55-870d0f8d948e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" Oct 29 22:30:59.755040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.755008 4952 generic.go:356] "Generic (PLEG): container finished" podID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerID="33be774a186283c70c55eaff9305d7a4638fe779317050bcd93994433177097a" exitCode=60 Oct 29 22:30:59.755538 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.755508 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" event={"ID":"2412c3a1-19b8-4bba-8399-590859b54a7e","Type":"ContainerDied","Data":"33be774a186283c70c55eaff9305d7a4638fe779317050bcd93994433177097a"} Oct 29 22:30:59.756383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.756361 4952 scope.go:122] "RemoveContainer" containerID="33be774a186283c70c55eaff9305d7a4638fe779317050bcd93994433177097a" Oct 29 22:30:59.757100 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:59.757059 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp_odh-ai-gateway-infra(2412c3a1-19b8-4bba-8399-590859b54a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" Oct 29 22:30:59.770487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.770439 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-maas-api-service-ca\") pod \"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a\" (UID: \"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a\") " Oct 29 22:30:59.770803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.770783 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-kube-api-access-hjz78\" (UniqueName: \"kubernetes.io/projected/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-kube-api-access-hjz78\") pod \"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a\" (UID: \"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a\") " Oct 29 22:30:59.795806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.795640 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2"] Oct 29 22:30:59.814557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.814501 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6"] Oct 29 22:30:59.818288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.816496 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-kube-api-access-hjz78" pod "6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" (UID: "6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a"). InnerVolumeSpecName "kube-api-access-hjz78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:59.844538 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.834390 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-maas-api-service-ca" pod "6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" (UID: "6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:59.893847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.879244 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:59.893847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.879277 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjz78\" (UniqueName: \"kubernetes.io/projected/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a-kube-api-access-hjz78\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:30:59.893847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.887484 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-istio-token\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.893847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.888275 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-envoy\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.893847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.892967 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f904684-fe8c-4e96-befb-229dbf30522f-istio-podinfo\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.941419 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.894737 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" event={"ID":"56e3163c-2722-49cb-a17d-11af94a932b5","Type":"ContainerDied","Data":"7f2dfdef2c79fd0b782e1b8bf62b3b00ac4ba6d39c4e3c7feddb678e30b4a4d8"} Oct 29 22:30:59.941419 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.898977 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7" Oct 29 22:30:59.945191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.945081 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9x7x\" (UniqueName: \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-kube-api-access-p9x7x\") pod \"e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:30:59.988607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.978150 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" event={"ID":"19785bf3-8a8c-4346-9a06-01123ccf2378","Type":"ContainerStarted","Data":"37b74906925a1924ea8e1e7074fd6ff238d6bef441c8547760fb9dbad5b6de99"} Oct 29 22:30:59.988607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:30:59.978671 4952 scope.go:122] "RemoveContainer" containerID="37b74906925a1924ea8e1e7074fd6ff238d6bef441c8547760fb9dbad5b6de99" Oct 29 22:30:59.988607 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:30:59.979169 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx_odh-ai-gateway-infra(19785bf3-8a8c-4346-9a06-01123ccf2378)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" Oct 29 22:31:00.039437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.039139 4952 generic.go:356] "Generic (PLEG): container finished" podID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerID="39a44c9b976951cb897411cad6f73d92ced50415cfe5dce075872236001561ac" exitCode=60 Oct 29 22:31:00.039437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.039240 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" event={"ID":"95df7b3a-e784-48ec-be58-f5620476c8f9","Type":"ContainerDied","Data":"39a44c9b976951cb897411cad6f73d92ced50415cfe5dce075872236001561ac"} Oct 29 22:31:00.045945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.045732 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" event={"ID":"e852730b-443d-4644-9a27-de432696d5b9","Type":"ContainerDied","Data":"a11edb7a324b393fa5a276fc1fa21b53997aac7e9abd762186ed77a50e30c689"} Oct 29 22:31:00.045945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.045850 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd" Oct 29 22:31:00.121768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.121733 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f"] Oct 29 22:31:00.182384 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.181809 4952 scope.go:122] "RemoveContainer" containerID="bb504afd4934409689149ffcea778a4ea69d427277c0854a21b47469d248c844" Oct 29 22:31:00.189597 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.189564 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:31:00.199972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.196395 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" event={"ID":"6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a","Type":"ContainerDied","Data":"bc6a1500ff41d13967a7c96f2ac533f45f54e27141bc202996d206546747dd22"} Oct 29 22:31:00.199972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.196525 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b" Oct 29 22:31:00.284090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.284052 4952 scope.go:122] "RemoveContainer" containerID="9bd7ae203fc6162aeca5df63e8b5bc635a6f0f696dd399ff42b2848f69fb6656" Oct 29 22:31:00.286265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.286245 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp"] Oct 29 22:31:00.340558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.340457 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq"] Oct 29 22:31:00.358052 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.357817 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp"] Oct 29 22:31:00.361279 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.361190 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh"] Oct 29 22:31:00.403494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.403461 4952 scope.go:122] "RemoveContainer" containerID="34aca47fc2d12fc113a80dfe3762355dc4f025c61d4828658396366d8773854f" Oct 29 22:31:00.446326 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.446249 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx"] Oct 29 22:31:00.541523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.539415 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5"] Oct 29 22:31:00.616343 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.616314 4952 scope.go:122] "RemoveContainer" containerID="08c2b108ddcd5a6005ddc18a71ab60b1e34be460cdb8ca7d2866f42611ecc953" Oct 29 22:31:00.617446 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.617417 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:00.761745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.761710 4952 scope.go:122] "RemoveContainer" containerID="fc8cfb037f235c2b7d8f578f20a998fc9232211e7d433dd7edc04998f51c9c8f" Oct 29 22:31:00.780309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.778351 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" path="/var/lib/kubelet/pods/49f4055f-a3b2-4146-a50a-888fb5d73a35/volumes" Oct 29 22:31:00.780309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.779073 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" path="/var/lib/kubelet/pods/dafdb0e0-ba9f-4e05-ad91-1482995b0466/volumes" Oct 29 22:31:00.802250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.802219 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" Oct 29 22:31:00.854840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.854757 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" Oct 29 22:31:00.879274 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.879177 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6"] Oct 29 22:31:00.895290 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.893059 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae173ee2-aa93-41b3-a205-7bf63dfdba83-kube-api-access-64j2x\" (UniqueName: \"kubernetes.io/projected/ae173ee2-aa93-41b3-a205-7bf63dfdba83-kube-api-access-64j2x\") pod \"ae173ee2-aa93-41b3-a205-7bf63dfdba83\" (UID: \"ae173ee2-aa93-41b3-a205-7bf63dfdba83\") " Oct 29 22:31:00.895290 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.893142 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae173ee2-aa93-41b3-a205-7bf63dfdba83-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae173ee2-aa93-41b3-a205-7bf63dfdba83-maas-api-service-ca\") pod \"ae173ee2-aa93-41b3-a205-7bf63dfdba83\" (UID: \"ae173ee2-aa93-41b3-a205-7bf63dfdba83\") " Oct 29 22:31:00.901953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.901890 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae173ee2-aa93-41b3-a205-7bf63dfdba83-maas-api-service-ca" pod "ae173ee2-aa93-41b3-a205-7bf63dfdba83" (UID: "ae173ee2-aa93-41b3-a205-7bf63dfdba83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:00.981148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.971025 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7"] Oct 29 22:31:00.981148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.971661 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae173ee2-aa93-41b3-a205-7bf63dfdba83-kube-api-access-64j2x" pod "ae173ee2-aa93-41b3-a205-7bf63dfdba83" (UID: "ae173ee2-aa93-41b3-a205-7bf63dfdba83"). InnerVolumeSpecName "kube-api-access-64j2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:00.981148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.981034 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c06a2c-29888550-gklx7"] Oct 29 22:31:00.997020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.995025 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-maas-api-service-ca\") pod \"c0e57d7c-7064-43f7-bec3-b92ae3aba56e\" (UID: \"c0e57d7c-7064-43f7-bec3-b92ae3aba56e\") " Oct 29 22:31:00.997020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.995142 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-kube-api-access-nwblc\" (UniqueName: \"kubernetes.io/projected/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-kube-api-access-nwblc\") pod \"c0e57d7c-7064-43f7-bec3-b92ae3aba56e\" (UID: \"c0e57d7c-7064-43f7-bec3-b92ae3aba56e\") " Oct 29 22:31:00.997020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.996312 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64j2x\" (UniqueName: \"kubernetes.io/projected/ae173ee2-aa93-41b3-a205-7bf63dfdba83-kube-api-access-64j2x\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:00.997020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.996342 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae173ee2-aa93-41b3-a205-7bf63dfdba83-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:00.997657 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:00.997600 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-maas-api-service-ca" pod "c0e57d7c-7064-43f7-bec3-b92ae3aba56e" (UID: "c0e57d7c-7064-43f7-bec3-b92ae3aba56e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:01.007386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.007095 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-kube-api-access-nwblc" pod "c0e57d7c-7064-43f7-bec3-b92ae3aba56e" (UID: "c0e57d7c-7064-43f7-bec3-b92ae3aba56e"). InnerVolumeSpecName "kube-api-access-nwblc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:01.040345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.039347 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b"] Oct 29 22:31:01.048345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.048276 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82db63-29888550-ph74b"] Oct 29 22:31:01.066372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.066308 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd"] Oct 29 22:31:01.078375 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.078293 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-92d984-29888550-cnhnd"] Oct 29 22:31:01.098580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.098171 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.098580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.098206 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwblc\" (UniqueName: \"kubernetes.io/projected/c0e57d7c-7064-43f7-bec3-b92ae3aba56e-kube-api-access-nwblc\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.106048 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.102282 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" Oct 29 22:31:01.106048 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.105473 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" Oct 29 22:31:01.110572 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.110546 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" Oct 29 22:31:01.115747 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.115082 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" Oct 29 22:31:01.126264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.125534 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" Oct 29 22:31:01.198962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.198890 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d92b54-ecdc-4dc5-842b-be8811d0a761-kube-api-access-z7rr8\" (UniqueName: \"kubernetes.io/projected/f9d92b54-ecdc-4dc5-842b-be8811d0a761-kube-api-access-z7rr8\") pod \"f9d92b54-ecdc-4dc5-842b-be8811d0a761\" (UID: \"f9d92b54-ecdc-4dc5-842b-be8811d0a761\") " Oct 29 22:31:01.199141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.199052 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95df7b3a-e784-48ec-be58-f5620476c8f9-kube-api-access-fss5q\" (UniqueName: \"kubernetes.io/projected/95df7b3a-e784-48ec-be58-f5620476c8f9-kube-api-access-fss5q\") pod \"95df7b3a-e784-48ec-be58-f5620476c8f9\" (UID: \"95df7b3a-e784-48ec-be58-f5620476c8f9\") " Oct 29 22:31:01.199141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.199114 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95df7b3a-e784-48ec-be58-f5620476c8f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df7b3a-e784-48ec-be58-f5620476c8f9-maas-api-service-ca\") pod \"95df7b3a-e784-48ec-be58-f5620476c8f9\" (UID: \"95df7b3a-e784-48ec-be58-f5620476c8f9\") " Oct 29 22:31:01.199265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.199148 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-maas-api-service-ca\") pod \"8dc0b51a-0307-4c90-b5ca-cd00ea927a16\" (UID: \"8dc0b51a-0307-4c90-b5ca-cd00ea927a16\") " Oct 29 22:31:01.199265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.199197 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-maas-api-service-ca\") pod \"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6\" (UID: \"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6\") " Oct 29 22:31:01.199265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.199242 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ea33cb-020d-415a-b518-37018182741a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ea33cb-020d-415a-b518-37018182741a-maas-api-service-ca\") pod \"06ea33cb-020d-415a-b518-37018182741a\" (UID: \"06ea33cb-020d-415a-b518-37018182741a\") " Oct 29 22:31:01.199455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.199291 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ea33cb-020d-415a-b518-37018182741a-kube-api-access-wvjrf\" (UniqueName: \"kubernetes.io/projected/06ea33cb-020d-415a-b518-37018182741a-kube-api-access-wvjrf\") pod \"06ea33cb-020d-415a-b518-37018182741a\" (UID: \"06ea33cb-020d-415a-b518-37018182741a\") " Oct 29 22:31:01.199455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.199405 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-kube-api-access-b22f6\" (UniqueName: \"kubernetes.io/projected/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-kube-api-access-b22f6\") pod \"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6\" (UID: \"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6\") " Oct 29 22:31:01.199569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.199531 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d92b54-ecdc-4dc5-842b-be8811d0a761-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d92b54-ecdc-4dc5-842b-be8811d0a761-maas-api-service-ca\") pod \"f9d92b54-ecdc-4dc5-842b-be8811d0a761\" (UID: \"f9d92b54-ecdc-4dc5-842b-be8811d0a761\") " Oct 29 22:31:01.199678 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.199656 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-kube-api-access-gk552\" (UniqueName: \"kubernetes.io/projected/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-kube-api-access-gk552\") pod \"8dc0b51a-0307-4c90-b5ca-cd00ea927a16\" (UID: \"8dc0b51a-0307-4c90-b5ca-cd00ea927a16\") " Oct 29 22:31:01.199872 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.199844 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-maas-api-service-ca" pod "0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" (UID: "0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:01.200255 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.200231 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95df7b3a-e784-48ec-be58-f5620476c8f9-maas-api-service-ca" pod "95df7b3a-e784-48ec-be58-f5620476c8f9" (UID: "95df7b3a-e784-48ec-be58-f5620476c8f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:01.200600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.200573 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-maas-api-service-ca" pod "8dc0b51a-0307-4c90-b5ca-cd00ea927a16" (UID: "8dc0b51a-0307-4c90-b5ca-cd00ea927a16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:01.200950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.200925 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d92b54-ecdc-4dc5-842b-be8811d0a761-maas-api-service-ca" pod "f9d92b54-ecdc-4dc5-842b-be8811d0a761" (UID: "f9d92b54-ecdc-4dc5-842b-be8811d0a761"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:01.201192 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.201171 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d92b54-ecdc-4dc5-842b-be8811d0a761-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.201308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.201294 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df7b3a-e784-48ec-be58-f5620476c8f9-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.201394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.201380 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.201479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.201467 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.204532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.204441 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d92b54-ecdc-4dc5-842b-be8811d0a761-kube-api-access-z7rr8" pod "f9d92b54-ecdc-4dc5-842b-be8811d0a761" (UID: "f9d92b54-ecdc-4dc5-842b-be8811d0a761"). InnerVolumeSpecName "kube-api-access-z7rr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:01.206990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.204956 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ea33cb-020d-415a-b518-37018182741a-maas-api-service-ca" pod "06ea33cb-020d-415a-b518-37018182741a" (UID: "06ea33cb-020d-415a-b518-37018182741a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:01.206990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.205695 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-kube-api-access-b22f6" pod "0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" (UID: "0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6"). InnerVolumeSpecName "kube-api-access-b22f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:01.206990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.205819 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ea33cb-020d-415a-b518-37018182741a-kube-api-access-wvjrf" pod "06ea33cb-020d-415a-b518-37018182741a" (UID: "06ea33cb-020d-415a-b518-37018182741a"). InnerVolumeSpecName "kube-api-access-wvjrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:01.206990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.205980 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-kube-api-access-gk552" pod "8dc0b51a-0307-4c90-b5ca-cd00ea927a16" (UID: "8dc0b51a-0307-4c90-b5ca-cd00ea927a16"). InnerVolumeSpecName "kube-api-access-gk552". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:01.212755 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.212727 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95df7b3a-e784-48ec-be58-f5620476c8f9-kube-api-access-fss5q" pod "95df7b3a-e784-48ec-be58-f5620476c8f9" (UID: "95df7b3a-e784-48ec-be58-f5620476c8f9"). InnerVolumeSpecName "kube-api-access-fss5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:01.243176 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.243096 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" Oct 29 22:31:01.260057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.247848 4952 generic.go:356] "Generic (PLEG): container finished" podID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerID="e7b336f990ffe4d1d9c941b08c31b416c86f239501de9eec0a8050828d7ba9c7" exitCode=60 Oct 29 22:31:01.260057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.250971 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd" event={"ID":"0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6","Type":"ContainerDied","Data":"04374893b4aa467948327197411456907c674786e693ff21e2d408e2a2d6648f"} Oct 29 22:31:01.260057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.251035 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" event={"ID":"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c","Type":"ContainerDied","Data":"e7b336f990ffe4d1d9c941b08c31b416c86f239501de9eec0a8050828d7ba9c7"} Oct 29 22:31:01.260057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.251068 4952 scope.go:122] "RemoveContainer" containerID="721e4697289f5ff92c2d80908b4d89ffb0803fcdf36ccc5542fbda683ffc5cbf" Oct 29 22:31:01.260057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.256227 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" event={"ID":"c0e57d7c-7064-43f7-bec3-b92ae3aba56e","Type":"ContainerDied","Data":"27c76fd655a0dc49e0513be476e7d1b058e30d06bff76f4e092ab57eae46b916"} Oct 29 22:31:01.260057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.256322 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh" Oct 29 22:31:01.273517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.271059 4952 generic.go:356] "Generic (PLEG): container finished" podID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerID="3a3fc7dc1edcdf1cc06a4c6a618e85f6f500aa6409f1e071d3f6084ae7b2fa20" exitCode=60 Oct 29 22:31:01.273517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.271164 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" event={"ID":"4c674559-f3d7-47cf-a264-aa41039dc61f","Type":"ContainerDied","Data":"3a3fc7dc1edcdf1cc06a4c6a618e85f6f500aa6409f1e071d3f6084ae7b2fa20"} Oct 29 22:31:01.278680 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.278647 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" event={"ID":"95df7b3a-e784-48ec-be58-f5620476c8f9","Type":"ContainerDied","Data":"91fcc01bec67188812a7aa259eee463552fcb2dfe93c6ad6f399d75b25cf10c1"} Oct 29 22:31:01.278862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.278695 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2" Oct 29 22:31:01.283835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.283804 4952 generic.go:356] "Generic (PLEG): container finished" podID="575c1d37-7dc0-4270-b023-192457eadac8" containerID="45d3b684dfd7917af74715459bec7596a9549450ad08e5b9cbc153ffab8a9db0" exitCode=60 Oct 29 22:31:01.284014 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.283883 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" event={"ID":"575c1d37-7dc0-4270-b023-192457eadac8","Type":"ContainerDied","Data":"45d3b684dfd7917af74715459bec7596a9549450ad08e5b9cbc153ffab8a9db0"} Oct 29 22:31:01.302020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.301617 4952 scope.go:122] "RemoveContainer" containerID="df5f8a94637863b2a3040fbe3fb330ef19b6aa89fa841e185e62cc6f8f0ed18c" Oct 29 22:31:01.304917 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.304791 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b22f6\" (UniqueName: \"kubernetes.io/projected/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6-kube-api-access-b22f6\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.304917 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.304819 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk552\" (UniqueName: \"kubernetes.io/projected/8dc0b51a-0307-4c90-b5ca-cd00ea927a16-kube-api-access-gk552\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.304917 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.304835 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7rr8\" (UniqueName: \"kubernetes.io/projected/f9d92b54-ecdc-4dc5-842b-be8811d0a761-kube-api-access-z7rr8\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.304917 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.304848 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fss5q\" (UniqueName: \"kubernetes.io/projected/95df7b3a-e784-48ec-be58-f5620476c8f9-kube-api-access-fss5q\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.304917 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.304862 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ea33cb-020d-415a-b518-37018182741a-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.304917 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.304877 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvjrf\" (UniqueName: \"kubernetes.io/projected/06ea33cb-020d-415a-b518-37018182741a-kube-api-access-wvjrf\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.338988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.336974 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" event={"ID":"8dc0b51a-0307-4c90-b5ca-cd00ea927a16","Type":"ContainerDied","Data":"628afa52d168ee7d7e8fad93cdf8311d579fe95957a53af6cee1b2e31923a124"} Oct 29 22:31:01.338988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.337209 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2" Oct 29 22:31:01.346793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.344491 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" event={"ID":"f9d92b54-ecdc-4dc5-842b-be8811d0a761","Type":"ContainerDied","Data":"57be6952df9daba0fa0cd286a4cf054e2180f9140fe81b7240081ad097311930"} Oct 29 22:31:01.346793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.344608 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f" Oct 29 22:31:01.350308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.350284 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt"] Oct 29 22:31:01.352249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.352225 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" event={"ID":"06ea33cb-020d-415a-b518-37018182741a","Type":"ContainerDied","Data":"3592766599a3e939d8f4f3ac92a587809af47603541326b452d834be4a7f1c3b"} Oct 29 22:31:01.352428 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.350772 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp" Oct 29 22:31:01.373251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.373204 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" event={"ID":"ae173ee2-aa93-41b3-a205-7bf63dfdba83","Type":"ContainerDied","Data":"df2da63a8dd51ca49613853a36abb97cf8db0dd272aad7b832ec412f18b1325d"} Oct 29 22:31:01.373500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.373478 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6" Oct 29 22:31:01.390480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.388981 4952 generic.go:356] "Generic (PLEG): container finished" podID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerID="37b74906925a1924ea8e1e7074fd6ff238d6bef441c8547760fb9dbad5b6de99" exitCode=60 Oct 29 22:31:01.390480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.389038 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" event={"ID":"19785bf3-8a8c-4346-9a06-01123ccf2378","Type":"ContainerDied","Data":"37b74906925a1924ea8e1e7074fd6ff238d6bef441c8547760fb9dbad5b6de99"} Oct 29 22:31:01.414550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.405917 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:31:01.414550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.406012 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:31:01.414550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.406056 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:31:01.456412 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.456382 4952 scope.go:122] "RemoveContainer" containerID="e9be1ffd81d2b088283f007d9a852814dfce1b69c9a2e0ad84a6383c5279ff75" Oct 29 22:31:01.503943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.495441 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh"] Oct 29 22:31:01.503943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.501690 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-50d4af-29888550-bbzsh"] Oct 29 22:31:01.542602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.532819 4952 scope.go:122] "RemoveContainer" containerID="12024f649c698ac098cae652cafa62a953dd18e8efe66ddcbf2419cb5706afc8" Oct 29 22:31:01.599642 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.598327 4952 scope.go:122] "RemoveContainer" containerID="39a44c9b976951cb897411cad6f73d92ced50415cfe5dce075872236001561ac" Oct 29 22:31:01.614071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.613744 4952 scope.go:122] "RemoveContainer" containerID="89e0790d52ed2e86b5723c4b7695fa1410f1eb11e823f77c6520c7b30bbb8347" Oct 29 22:31:01.665092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.665057 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd"] Oct 29 22:31:01.669922 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.669864 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0412c3-29888550-jb8fd"] Oct 29 22:31:01.686591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.686555 4952 scope.go:122] "RemoveContainer" containerID="b91ce6025b05007bcf2a8eab36c82a5c476ea135f7ace7729cb7b67e475dc6a4" Oct 29 22:31:01.703629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.703564 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" Oct 29 22:31:01.713755 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.713629 4952 scope.go:122] "RemoveContainer" containerID="099488ab71fb43fa5a9b11cd828363973294ef70f918dfdc1773f67538aa9ed3" Oct 29 22:31:01.788167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.784141 4952 scope.go:122] "RemoveContainer" containerID="71c16d991ef0ca9a14e1ecf795fcd0b82866b0cc84a39797cdc6b1758d5e56fc" Oct 29 22:31:01.810516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.807763 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2"] Oct 29 22:31:01.815439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.815399 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c674559-f3d7-47cf-a264-aa41039dc61f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c674559-f3d7-47cf-a264-aa41039dc61f-maas-api-service-ca\") pod \"4c674559-f3d7-47cf-a264-aa41039dc61f\" (UID: \"4c674559-f3d7-47cf-a264-aa41039dc61f\") " Oct 29 22:31:01.815831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.815803 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c674559-f3d7-47cf-a264-aa41039dc61f-kube-api-access-jz665\" (UniqueName: \"kubernetes.io/projected/4c674559-f3d7-47cf-a264-aa41039dc61f-kube-api-access-jz665\") pod \"4c674559-f3d7-47cf-a264-aa41039dc61f\" (UID: \"4c674559-f3d7-47cf-a264-aa41039dc61f\") " Oct 29 22:31:01.816969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.816935 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c674559-f3d7-47cf-a264-aa41039dc61f-maas-api-service-ca" pod "4c674559-f3d7-47cf-a264-aa41039dc61f" (UID: "4c674559-f3d7-47cf-a264-aa41039dc61f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:01.828018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.827428 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" Oct 29 22:31:01.834847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.834821 4952 scope.go:122] "RemoveContainer" containerID="48cd103a887360d61e37fa3c83836caa26fbc7dcd042bd99296cf164b151f827" Oct 29 22:31:01.835835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.835815 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" Oct 29 22:31:01.839830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.839756 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3b067-29888550-bpgb2"] Oct 29 22:31:01.854928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.854875 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c674559-f3d7-47cf-a264-aa41039dc61f-kube-api-access-jz665" pod "4c674559-f3d7-47cf-a264-aa41039dc61f" (UID: "4c674559-f3d7-47cf-a264-aa41039dc61f"). InnerVolumeSpecName "kube-api-access-jz665". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:01.855386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.855245 4952 scope.go:122] "RemoveContainer" containerID="fcc03dfac458137483b9d19f6ece49a9e9dfc07d32487c5bd1cbe93dcb19e39f" Oct 29 22:31:01.868379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.868348 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp"] Oct 29 22:31:01.878979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.878727 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-adfb9d-29888550-lfhbp"] Oct 29 22:31:01.926698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.926662 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-kube-api-access-tdfmz\" (UniqueName: \"kubernetes.io/projected/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-kube-api-access-tdfmz\") pod \"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c\" (UID: \"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c\") " Oct 29 22:31:01.927221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.927199 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575c1d37-7dc0-4270-b023-192457eadac8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575c1d37-7dc0-4270-b023-192457eadac8-maas-api-service-ca\") pod \"575c1d37-7dc0-4270-b023-192457eadac8\" (UID: \"575c1d37-7dc0-4270-b023-192457eadac8\") " Oct 29 22:31:01.927470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.927455 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-maas-api-service-ca\") pod \"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c\" (UID: \"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c\") " Oct 29 22:31:01.927600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.927586 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575c1d37-7dc0-4270-b023-192457eadac8-kube-api-access-9ptdx\" (UniqueName: \"kubernetes.io/projected/575c1d37-7dc0-4270-b023-192457eadac8-kube-api-access-9ptdx\") pod \"575c1d37-7dc0-4270-b023-192457eadac8\" (UID: \"575c1d37-7dc0-4270-b023-192457eadac8\") " Oct 29 22:31:01.928517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.928494 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz665\" (UniqueName: \"kubernetes.io/projected/4c674559-f3d7-47cf-a264-aa41039dc61f-kube-api-access-jz665\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.928651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.928638 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c674559-f3d7-47cf-a264-aa41039dc61f-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:01.930644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.930357 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575c1d37-7dc0-4270-b023-192457eadac8-maas-api-service-ca" pod "575c1d37-7dc0-4270-b023-192457eadac8" (UID: "575c1d37-7dc0-4270-b023-192457eadac8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:01.933534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.933506 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575c1d37-7dc0-4270-b023-192457eadac8-kube-api-access-9ptdx" pod "575c1d37-7dc0-4270-b023-192457eadac8" (UID: "575c1d37-7dc0-4270-b023-192457eadac8"). InnerVolumeSpecName "kube-api-access-9ptdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:01.951180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.933659 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-maas-api-service-ca" pod "20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" (UID: "20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:01.951180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.950985 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-kube-api-access-tdfmz" pod "20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" (UID: "20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c"). InnerVolumeSpecName "kube-api-access-tdfmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:01.985807 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:01.985742 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2"] Oct 29 22:31:02.019297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.019229 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-76c1b7-29888550-hhdl2"] Oct 29 22:31:02.036735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.034979 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ptdx\" (UniqueName: \"kubernetes.io/projected/575c1d37-7dc0-4270-b023-192457eadac8-kube-api-access-9ptdx\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.036735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.035012 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdfmz\" (UniqueName: \"kubernetes.io/projected/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-kube-api-access-tdfmz\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.036735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.035027 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575c1d37-7dc0-4270-b023-192457eadac8-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.036735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.035041 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.053199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.053159 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6"] Oct 29 22:31:02.059759 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.058898 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888550-9v2m6"] Oct 29 22:31:02.075102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.075072 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" Oct 29 22:31:02.079264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.078034 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" Oct 29 22:31:02.087029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.086707 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" Oct 29 22:31:02.130840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.130412 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f"] Oct 29 22:31:02.137672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.136960 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19785bf3-8a8c-4346-9a06-01123ccf2378-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19785bf3-8a8c-4346-9a06-01123ccf2378-maas-api-service-ca\") pod \"19785bf3-8a8c-4346-9a06-01123ccf2378\" (UID: \"19785bf3-8a8c-4346-9a06-01123ccf2378\") " Oct 29 22:31:02.137817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.137757 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29c2f2-29888550-vnr7f"] Oct 29 22:31:02.139025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.138996 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19785bf3-8a8c-4346-9a06-01123ccf2378-maas-api-service-ca" pod "19785bf3-8a8c-4346-9a06-01123ccf2378" (UID: "19785bf3-8a8c-4346-9a06-01123ccf2378"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:02.139177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.139009 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2412c3a1-19b8-4bba-8399-590859b54a7e-kube-api-access-2l9tj\" (UniqueName: \"kubernetes.io/projected/2412c3a1-19b8-4bba-8399-590859b54a7e-kube-api-access-2l9tj\") pod \"2412c3a1-19b8-4bba-8399-590859b54a7e\" (UID: \"2412c3a1-19b8-4bba-8399-590859b54a7e\") " Oct 29 22:31:02.139358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.139340 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d669ddd4-8a30-4078-9c55-870d0f8d948e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d669ddd4-8a30-4078-9c55-870d0f8d948e-maas-api-service-ca\") pod \"d669ddd4-8a30-4078-9c55-870d0f8d948e\" (UID: \"d669ddd4-8a30-4078-9c55-870d0f8d948e\") " Oct 29 22:31:02.139580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.139563 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2412c3a1-19b8-4bba-8399-590859b54a7e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2412c3a1-19b8-4bba-8399-590859b54a7e-maas-api-service-ca\") pod \"2412c3a1-19b8-4bba-8399-590859b54a7e\" (UID: \"2412c3a1-19b8-4bba-8399-590859b54a7e\") " Oct 29 22:31:02.139720 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.139705 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d669ddd4-8a30-4078-9c55-870d0f8d948e-kube-api-access-qg27c\" (UniqueName: \"kubernetes.io/projected/d669ddd4-8a30-4078-9c55-870d0f8d948e-kube-api-access-qg27c\") pod \"d669ddd4-8a30-4078-9c55-870d0f8d948e\" (UID: \"d669ddd4-8a30-4078-9c55-870d0f8d948e\") " Oct 29 22:31:02.139953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.139937 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19785bf3-8a8c-4346-9a06-01123ccf2378-kube-api-access-8dzdd\" (UniqueName: \"kubernetes.io/projected/19785bf3-8a8c-4346-9a06-01123ccf2378-kube-api-access-8dzdd\") pod \"19785bf3-8a8c-4346-9a06-01123ccf2378\" (UID: \"19785bf3-8a8c-4346-9a06-01123ccf2378\") " Oct 29 22:31:02.140621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.139969 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d669ddd4-8a30-4078-9c55-870d0f8d948e-maas-api-service-ca" pod "d669ddd4-8a30-4078-9c55-870d0f8d948e" (UID: "d669ddd4-8a30-4078-9c55-870d0f8d948e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:02.140621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.140282 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2412c3a1-19b8-4bba-8399-590859b54a7e-maas-api-service-ca" pod "2412c3a1-19b8-4bba-8399-590859b54a7e" (UID: "2412c3a1-19b8-4bba-8399-590859b54a7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:02.141168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.141147 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d669ddd4-8a30-4078-9c55-870d0f8d948e-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.141280 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.141267 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2412c3a1-19b8-4bba-8399-590859b54a7e-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.141405 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.141393 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19785bf3-8a8c-4346-9a06-01123ccf2378-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.142397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.141987 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2412c3a1-19b8-4bba-8399-590859b54a7e-kube-api-access-2l9tj" pod "2412c3a1-19b8-4bba-8399-590859b54a7e" (UID: "2412c3a1-19b8-4bba-8399-590859b54a7e"). InnerVolumeSpecName "kube-api-access-2l9tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:02.144928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.144215 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19785bf3-8a8c-4346-9a06-01123ccf2378-kube-api-access-8dzdd" pod "19785bf3-8a8c-4346-9a06-01123ccf2378" (UID: "19785bf3-8a8c-4346-9a06-01123ccf2378"). InnerVolumeSpecName "kube-api-access-8dzdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:02.146661 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.145427 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d669ddd4-8a30-4078-9c55-870d0f8d948e-kube-api-access-qg27c" pod "d669ddd4-8a30-4078-9c55-870d0f8d948e" (UID: "d669ddd4-8a30-4078-9c55-870d0f8d948e"). InnerVolumeSpecName "kube-api-access-qg27c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:02.247929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.244684 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg27c\" (UniqueName: \"kubernetes.io/projected/d669ddd4-8a30-4078-9c55-870d0f8d948e-kube-api-access-qg27c\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.247929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.244722 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dzdd\" (UniqueName: \"kubernetes.io/projected/19785bf3-8a8c-4346-9a06-01123ccf2378-kube-api-access-8dzdd\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.247929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.244739 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l9tj\" (UniqueName: \"kubernetes.io/projected/2412c3a1-19b8-4bba-8399-590859b54a7e-kube-api-access-2l9tj\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:02.534664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.533194 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" event={"ID":"d669ddd4-8a30-4078-9c55-870d0f8d948e","Type":"ContainerDied","Data":"151e7856ca966adc9aebe7e6ce1c818ec155bd163665e7f0f27889994e99efbd"} Oct 29 22:31:02.535007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.534984 4952 scope.go:122] "RemoveContainer" containerID="1c25be5be81f9dd4afc0a2168eb46c1f519841aeeef785cc88a3eed3fc0035a5" Oct 29 22:31:02.535226 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.535211 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq" Oct 29 22:31:02.542317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.542293 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" Oct 29 22:31:02.551248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.551154 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp" event={"ID":"2412c3a1-19b8-4bba-8399-590859b54a7e","Type":"ContainerDied","Data":"75292a480f3394c1da645c4b046e6c3e5cce9cdb77167ceb9f89045227274418"} Oct 29 22:31:02.559325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.559291 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" event={"ID":"9f904684-fe8c-4e96-befb-229dbf30522f","Type":"ContainerStarted","Data":"b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7"} Oct 29 22:31:02.563190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.563158 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" event={"ID":"9f904684-fe8c-4e96-befb-229dbf30522f","Type":"ContainerStarted","Data":"13d613454dd90e1d7210aaf712c52ee1649bef1c9d85e5070dda7fa7bede612d"} Oct 29 22:31:02.563642 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.563612 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" event={"ID":"19785bf3-8a8c-4346-9a06-01123ccf2378","Type":"ContainerDied","Data":"da994ecffc8013a95be077c0ea22bff1b6f2763e89e867e12d4202c405282e1d"} Oct 29 22:31:02.567800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.566585 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx" Oct 29 22:31:02.575414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.574759 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" event={"ID":"20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c","Type":"ContainerDied","Data":"b2b3618c1163c41f1b37a52ba9015cfa8da052a85875bf6ffc5eaa9f9c6e7abb"} Oct 29 22:31:02.575414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.574853 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5" Oct 29 22:31:02.581530 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.581502 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" event={"ID":"4c674559-f3d7-47cf-a264-aa41039dc61f","Type":"ContainerDied","Data":"9ce8d73092f26e8406d908898ee39dfff9e691a14e5abe63dc5b59492691007e"} Oct 29 22:31:02.582463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.582441 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh" Oct 29 22:31:02.593077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.593047 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" event={"ID":"575c1d37-7dc0-4270-b023-192457eadac8","Type":"ContainerDied","Data":"7ffcfe62d745e2b7f40b99228a47f7ba988828921c4fd09dcfc203af9914e843"} Oct 29 22:31:02.593331 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.593313 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6" Oct 29 22:31:02.658368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.657630 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" podStartSLOduration=3.657608947 podStartE2EDuration="3.657608947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:02.651695019 +0000 UTC m=+2928.126283418" watchObservedRunningTime="2026-10-29 22:31:02.657608947 +0000 UTC m=+2928.132197368" Oct 29 22:31:02.660211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.660157 4952 scope.go:122] "RemoveContainer" containerID="33be774a186283c70c55eaff9305d7a4638fe779317050bcd93994433177097a" Oct 29 22:31:02.667393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.667362 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ea33cb-020d-415a-b518-37018182741a" path="/var/lib/kubelet/pods/06ea33cb-020d-415a-b518-37018182741a/volumes" Oct 29 22:31:02.667982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.667961 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" path="/var/lib/kubelet/pods/0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6/volumes" Oct 29 22:31:02.668810 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.668789 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" path="/var/lib/kubelet/pods/56e3163c-2722-49cb-a17d-11af94a932b5/volumes" Oct 29 22:31:02.669407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.669386 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" path="/var/lib/kubelet/pods/6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a/volumes" Oct 29 22:31:02.669877 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.669857 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" path="/var/lib/kubelet/pods/8dc0b51a-0307-4c90-b5ca-cd00ea927a16/volumes" Oct 29 22:31:02.670726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.670706 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" path="/var/lib/kubelet/pods/95df7b3a-e784-48ec-be58-f5620476c8f9/volumes" Oct 29 22:31:02.671420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.671400 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" path="/var/lib/kubelet/pods/ae173ee2-aa93-41b3-a205-7bf63dfdba83/volumes" Oct 29 22:31:02.671950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.671932 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" path="/var/lib/kubelet/pods/c0e57d7c-7064-43f7-bec3-b92ae3aba56e/volumes" Oct 29 22:31:02.672697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.672676 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e852730b-443d-4644-9a27-de432696d5b9" path="/var/lib/kubelet/pods/e852730b-443d-4644-9a27-de432696d5b9/volumes" Oct 29 22:31:02.673248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.673228 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" path="/var/lib/kubelet/pods/f9d92b54-ecdc-4dc5-842b-be8811d0a761/volumes" Oct 29 22:31:02.679446 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.679426 4952 scope.go:122] "RemoveContainer" containerID="37b74906925a1924ea8e1e7074fd6ff238d6bef441c8547760fb9dbad5b6de99" Oct 29 22:31:02.709823 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.709679 4952 scope.go:122] "RemoveContainer" containerID="e7b336f990ffe4d1d9c941b08c31b416c86f239501de9eec0a8050828d7ba9c7" Oct 29 22:31:02.725634 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.725605 4952 scope.go:122] "RemoveContainer" containerID="3a3fc7dc1edcdf1cc06a4c6a618e85f6f500aa6409f1e071d3f6084ae7b2fa20" Oct 29 22:31:02.763146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.763090 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx"] Oct 29 22:31:02.828945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.823200 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-675bf4-29888550-fn4bx"] Oct 29 22:31:02.832941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.832467 4952 scope.go:122] "RemoveContainer" containerID="45d3b684dfd7917af74715459bec7596a9549450ad08e5b9cbc153ffab8a9db0" Oct 29 22:31:02.931424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.931391 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6"] Oct 29 22:31:02.937701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.937665 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:02.958974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:02.958214 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8d569e-29888550-cmmn6"] Oct 29 22:31:03.031930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.030839 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp"] Oct 29 22:31:03.053267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.053180 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cea94a-29888550-jgssp"] Oct 29 22:31:03.057549 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.057504 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq"] Oct 29 22:31:03.063447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.063418 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c1015-29888550-rlklq"] Oct 29 22:31:03.067862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.067837 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5"] Oct 29 22:31:03.072617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.072592 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-797e0c-29888550-xl8f5"] Oct 29 22:31:03.084348 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.084320 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh"] Oct 29 22:31:03.088329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.088305 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-hmrdh"] Oct 29 22:31:03.192781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.192746 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:31:03.199627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.199597 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:31:03.616751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.616724 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:31:03.617672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:03.617645 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:31:04.650410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:04.650375 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" path="/var/lib/kubelet/pods/19785bf3-8a8c-4346-9a06-01123ccf2378/volumes" Oct 29 22:31:04.651070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:04.650831 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" path="/var/lib/kubelet/pods/20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c/volumes" Oct 29 22:31:04.651403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:04.651385 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" path="/var/lib/kubelet/pods/2412c3a1-19b8-4bba-8399-590859b54a7e/volumes" Oct 29 22:31:04.652146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:04.652128 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" path="/var/lib/kubelet/pods/4c674559-f3d7-47cf-a264-aa41039dc61f/volumes" Oct 29 22:31:04.652548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:04.652533 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575c1d37-7dc0-4270-b023-192457eadac8" path="/var/lib/kubelet/pods/575c1d37-7dc0-4270-b023-192457eadac8/volumes" Oct 29 22:31:04.652984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:04.652968 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" path="/var/lib/kubelet/pods/d669ddd4-8a30-4078-9c55-870d0f8d948e/volumes" Oct 29 22:31:05.032168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:05.032008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:05.212374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:05.212340 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:05.376317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:05.376276 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:06.079172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:06.079135 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:06.786597 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:06.786550 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:06.959566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:06.959523 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:07.099637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.099605 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f"] Oct 29 22:31:07.100730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100712 4952 cpu_manager.go:414] "containerMap: removing container" podUID="575c1d37-7dc0-4270-b023-192457eadac8" containerName="cleanup" Oct 29 22:31:07.100786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100734 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575c1d37-7dc0-4270-b023-192457eadac8" containerName="cleanup" Oct 29 22:31:07.100786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100749 4952 cpu_manager.go:414] "containerMap: removing container" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerName="cleanup" Oct 29 22:31:07.100786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100758 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerName="cleanup" Oct 29 22:31:07.100786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100771 4952 cpu_manager.go:414] "containerMap: removing container" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerName="cleanup" Oct 29 22:31:07.100786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100778 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100789 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e852730b-443d-4644-9a27-de432696d5b9" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100798 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e852730b-443d-4644-9a27-de432696d5b9" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100817 4952 cpu_manager.go:414] "containerMap: removing container" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100824 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100838 4952 cpu_manager.go:414] "containerMap: removing container" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100845 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafdb0e0-ba9f-4e05-ad91-1482995b0466" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100862 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100869 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100884 4952 cpu_manager.go:414] "containerMap: removing container" podUID="06ea33cb-020d-415a-b518-37018182741a" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100891 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ea33cb-020d-415a-b518-37018182741a" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100900 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100920 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100931 4952 cpu_manager.go:414] "containerMap: removing container" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100937 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100953 4952 cpu_manager.go:414] "containerMap: removing container" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100960 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100971 4952 cpu_manager.go:414] "containerMap: removing container" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerName="cleanup" Oct 29 22:31:07.100971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100978 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100988 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.100994 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101004 4952 cpu_manager.go:414] "containerMap: removing container" podUID="575c1d37-7dc0-4270-b023-192457eadac8" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101012 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575c1d37-7dc0-4270-b023-192457eadac8" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101026 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101032 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101042 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101049 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101059 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101065 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101079 4952 cpu_manager.go:414] "containerMap: removing container" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101085 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101094 4952 cpu_manager.go:414] "containerMap: removing container" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101100 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101116 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101123 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101132 4952 cpu_manager.go:414] "containerMap: removing container" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101138 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101146 4952 cpu_manager.go:414] "containerMap: removing container" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101152 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101172 4952 cpu_manager.go:414] "containerMap: removing container" podUID="575c1d37-7dc0-4270-b023-192457eadac8" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101179 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575c1d37-7dc0-4270-b023-192457eadac8" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101190 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101196 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101207 4952 cpu_manager.go:414] "containerMap: removing container" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101214 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101222 4952 cpu_manager.go:414] "containerMap: removing container" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101228 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101242 4952 cpu_manager.go:414] "containerMap: removing container" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101248 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101257 4952 cpu_manager.go:414] "containerMap: removing container" podUID="06ea33cb-020d-415a-b518-37018182741a" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101264 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ea33cb-020d-415a-b518-37018182741a" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101275 4952 cpu_manager.go:414] "containerMap: removing container" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101283 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101301 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101307 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101320 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101327 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101336 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101342 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101356 4952 cpu_manager.go:414] "containerMap: removing container" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101362 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101372 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101379 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101595 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101608 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101617 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101623 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101631 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ea33cb-020d-415a-b518-37018182741a" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101637 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101643 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f4055f-a3b2-4146-a50a-888fb5d73a35" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101651 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101656 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101662 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="575c1d37-7dc0-4270-b023-192457eadac8" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101668 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101673 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101678 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101683 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101690 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101695 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101703 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101709 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101714 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e3163c-2722-49cb-a17d-11af94a932b5" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101722 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e852730b-443d-4644-9a27-de432696d5b9" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101729 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerName="cleanup" Oct 29 22:31:07.101690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101735 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="575c1d37-7dc0-4270-b023-192457eadac8" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101741 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101747 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101754 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101759 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ea33cb-020d-415a-b518-37018182741a" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101766 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101772 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101777 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101783 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101787 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101794 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101800 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101805 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101810 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101816 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerName="cleanup" Oct 29 22:31:07.104540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.101823 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2a80c8-d0b1-4286-9fb5-d828d64cc7f6" containerName="cleanup" Oct 29 22:31:07.110707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.110592 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:31:07.149264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.149095 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:07.165376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.165239 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-26abe8-dockercfg-lp275\"" Oct 29 22:31:07.165376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.165332 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-26abe8\"" Oct 29 22:31:07.166371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.166337 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f"] Oct 29 22:31:07.371112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.352384 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbv7h\" (UniqueName: \"kubernetes.io/projected/46cb18bb-e422-46b6-bbea-0881c23bcd7f-kube-api-access-pbv7h\") pod \"payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f\" (UID: \"46cb18bb-e422-46b6-bbea-0881c23bcd7f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:31:07.371112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.352518 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46cb18bb-e422-46b6-bbea-0881c23bcd7f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f\" (UID: \"46cb18bb-e422-46b6-bbea-0881c23bcd7f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:31:07.489928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.483473 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46cb18bb-e422-46b6-bbea-0881c23bcd7f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f\" (UID: \"46cb18bb-e422-46b6-bbea-0881c23bcd7f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:31:07.518924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.497978 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46cb18bb-e422-46b6-bbea-0881c23bcd7f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f\" (UID: \"46cb18bb-e422-46b6-bbea-0881c23bcd7f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:31:07.530932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.519857 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbv7h\" (UniqueName: \"kubernetes.io/projected/46cb18bb-e422-46b6-bbea-0881c23bcd7f-kube-api-access-pbv7h\") pod \"payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f\" (UID: \"46cb18bb-e422-46b6-bbea-0881c23bcd7f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:31:07.567982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.559454 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbv7h\" (UniqueName: \"kubernetes.io/projected/46cb18bb-e422-46b6-bbea-0881c23bcd7f-kube-api-access-pbv7h\") pod \"payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f\" (UID: \"46cb18bb-e422-46b6-bbea-0881c23bcd7f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:31:07.778625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.778545 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:31:07.919702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:07.919656 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:08.099660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:08.099628 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f"] Oct 29 22:31:08.100100 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:31:08.100062 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cb18bb_e422_46b6_bbea_0881c23bcd7f.slice/crio-1a44f31e5f9063e9c46a80ac18d2e55f04373a48fdc7d507884199301ae4e040 WatchSource:0}: Error finding container 1a44f31e5f9063e9c46a80ac18d2e55f04373a48fdc7d507884199301ae4e040: Status 404 returned error can't find the container with id 1a44f31e5f9063e9c46a80ac18d2e55f04373a48fdc7d507884199301ae4e040 Oct 29 22:31:08.647961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:08.641683 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:08.748929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:08.748667 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" event={"ID":"46cb18bb-e422-46b6-bbea-0881c23bcd7f","Type":"ContainerStarted","Data":"27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04"} Oct 29 22:31:08.748929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:08.748749 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" event={"ID":"46cb18bb-e422-46b6-bbea-0881c23bcd7f","Type":"ContainerStarted","Data":"1a44f31e5f9063e9c46a80ac18d2e55f04373a48fdc7d507884199301ae4e040"} Oct 29 22:31:08.748929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:08.748781 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:31:08.832218 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:08.822569 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" podStartSLOduration=1.822546318 podStartE2EDuration="1.822546318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:08.813523003 +0000 UTC m=+2934.288111447" watchObservedRunningTime="2026-10-29 22:31:08.822546318 +0000 UTC m=+2934.297134737" Oct 29 22:31:09.113334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:09.113299 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:09.338268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:09.338224 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:09.515192 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:09.515099 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:09.684831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:09.684795 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:09.936988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:09.936212 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:10.525394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:10.525356 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:10.701853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:10.701812 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:11.661293 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:11.661256 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:11.710054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:11.710023 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:31:12.020541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.020455 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:12.200072 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.200032 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:12.369540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.369473 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:12.465994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.465957 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6"] Oct 29 22:31:12.474927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.467360 4952 cpu_manager.go:414] "containerMap: removing container" podUID="06ea33cb-020d-415a-b518-37018182741a" containerName="cleanup" Oct 29 22:31:12.476996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.476970 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ea33cb-020d-415a-b518-37018182741a" containerName="cleanup" Oct 29 22:31:12.477206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.477192 4952 cpu_manager.go:414] "containerMap: removing container" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerName="cleanup" Oct 29 22:31:12.482926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.477283 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerName="cleanup" Oct 29 22:31:12.483167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.483146 4952 cpu_manager.go:414] "containerMap: removing container" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerName="cleanup" Oct 29 22:31:12.483273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.483261 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerName="cleanup" Oct 29 22:31:12.483364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.483353 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerName="cleanup" Oct 29 22:31:12.483444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.483433 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerName="cleanup" Oct 29 22:31:12.483535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.483525 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerName="cleanup" Oct 29 22:31:12.483627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.483616 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c674559-f3d7-47cf-a264-aa41039dc61f" containerName="cleanup" Oct 29 22:31:12.483738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.483726 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerName="cleanup" Oct 29 22:31:12.483821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.483810 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae173ee2-aa93-41b3-a205-7bf63dfdba83" containerName="cleanup" Oct 29 22:31:12.483942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.483929 4952 cpu_manager.go:414] "containerMap: removing container" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerName="cleanup" Oct 29 22:31:12.484030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.484018 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c15e30-5f6d-49c4-b8dd-f8a5c3d9244c" containerName="cleanup" Oct 29 22:31:12.484165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.484153 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerName="cleanup" Oct 29 22:31:12.499924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.484243 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerName="cleanup" Oct 29 22:31:12.501595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.501574 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerName="cleanup" Oct 29 22:31:12.503137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.503113 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad0a254-5a2a-4bd2-9f36-a7fe14591a2a" containerName="cleanup" Oct 29 22:31:12.503300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.503285 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerName="cleanup" Oct 29 22:31:12.503390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.503379 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerName="cleanup" Oct 29 22:31:12.503487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.503477 4952 cpu_manager.go:414] "containerMap: removing container" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerName="cleanup" Oct 29 22:31:12.503575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.503564 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19785bf3-8a8c-4346-9a06-01123ccf2378" containerName="cleanup" Oct 29 22:31:12.503957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.503937 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="575c1d37-7dc0-4270-b023-192457eadac8" containerName="cleanup" Oct 29 22:31:12.504080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.504068 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ea33cb-020d-415a-b518-37018182741a" containerName="cleanup" Oct 29 22:31:12.504197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.504186 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d669ddd4-8a30-4078-9c55-870d0f8d948e" containerName="cleanup" Oct 29 22:31:12.504301 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.504291 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d92b54-ecdc-4dc5-842b-be8811d0a761" containerName="cleanup" Oct 29 22:31:12.504434 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.504423 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="95df7b3a-e784-48ec-be58-f5620476c8f9" containerName="cleanup" Oct 29 22:31:12.504527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.504517 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc0b51a-0307-4c90-b5ca-cd00ea927a16" containerName="cleanup" Oct 29 22:31:12.504610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.504600 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e57d7c-7064-43f7-bec3-b92ae3aba56e" containerName="cleanup" Oct 29 22:31:12.504693 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.504684 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="2412c3a1-19b8-4bba-8399-590859b54a7e" containerName="cleanup" Oct 29 22:31:12.527273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.527243 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6"] Oct 29 22:31:12.527608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.527589 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:12.579930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.567445 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"payload-processing-external-model-plugins-e2e-extmod-a-749f72\"" Oct 29 22:31:12.579930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.568202 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"agc-cred-904a97a988a8\"" Oct 29 22:31:12.579930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.568295 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"routing-overlay\"" Oct 29 22:31:12.579930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.568391 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"kube-root-ca.crt\"" Oct 29 22:31:12.579930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.568445 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"openshift-service-ca.crt\"" Oct 29 22:31:12.623399 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.623313 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"payload-processing-external-model-e2e--d928d360-dockercfg-gp58q\"" Oct 29 22:31:12.863929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.838080 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:12.863929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.838237 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:12.863929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.838422 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9g9m\" (UniqueName: \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-kube-api-access-j9g9m\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:12.863929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.838476 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:12.916186 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.915340 4952 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podb03ee8fc-3424-4232-85ad-96a512607f60"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podb03ee8fc-3424-4232-85ad-96a512607f60] : Timed out while waiting for systemd to remove kubepods-burstable-podb03ee8fc_3424_4232_85ad_96a512607f60.slice" Oct 29 22:31:12.916186 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:31:12.915389 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable podb03ee8fc-3424-4232-85ad-96a512607f60] : unable to destroy cgroup paths for cgroup [kubepods burstable podb03ee8fc-3424-4232-85ad-96a512607f60] : Timed out while waiting for systemd to remove kubepods-burstable-podb03ee8fc_3424_4232_85ad_96a512607f60.slice" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" Oct 29 22:31:12.954994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.941852 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:12.954994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.952758 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:12.985207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.974108 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:12.985207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.974379 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9g9m\" (UniqueName: \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-kube-api-access-j9g9m\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:12.985207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.974438 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:12.985207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:12.975532 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:13.112923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.109249 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:13.125931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.123850 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9g9m\" (UniqueName: \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-kube-api-access-j9g9m\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:13.232502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.232418 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:13.367372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.367338 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7"] Oct 29 22:31:13.373640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.373605 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.395315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.395281 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-749f72\"/\"agc-cred-c2d72bb260f7\"" Oct 29 22:31:13.409192 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.409154 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7"] Oct 29 22:31:13.518468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.518387 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xg42\" (UniqueName: \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-kube-api-access-5xg42\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.518645 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.518571 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.518723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.518702 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.518785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.518735 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.553691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.553658 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6"] Oct 29 22:31:13.559413 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:31:13.559378 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4ab525_2e90_431a_b43c_0dfbea69b111.slice/crio-99994fc1aa721894c034b702cf506193966e82daf67e6e00dc5508acf75d5aa2 WatchSource:0}: Error finding container 99994fc1aa721894c034b702cf506193966e82daf67e6e00dc5508acf75d5aa2: Status 404 returned error can't find the container with id 99994fc1aa721894c034b702cf506193966e82daf67e6e00dc5508acf75d5aa2 Oct 29 22:31:13.619959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.619930 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xg42\" (UniqueName: \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-kube-api-access-5xg42\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.620099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.620053 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.620180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.620165 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.620260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.620212 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.621157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.621117 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.621797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.621774 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.623211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.623187 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.666979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.666362 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xg42\" (UniqueName: \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-kube-api-access-5xg42\") pod \"payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.711930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.706873 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:13.751443 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:13.751405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:14.042461 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:14.042432 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7"] Oct 29 22:31:14.043088 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:31:14.043055 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81383129_aa94_4349_a1c2_3a3a9240be1b.slice/crio-8fe4bb09fbe13113444d7510cc3ba7140bd1e557fd719c3ba08a5857eebbbcd9 WatchSource:0}: Error finding container 8fe4bb09fbe13113444d7510cc3ba7140bd1e557fd719c3ba08a5857eebbbcd9: Status 404 returned error can't find the container with id 8fe4bb09fbe13113444d7510cc3ba7140bd1e557fd719c3ba08a5857eebbbcd9 Oct 29 22:31:14.137836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:14.137787 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" event={"ID":"81383129-aa94-4349-a1c2-3a3a9240be1b","Type":"ContainerStarted","Data":"8fe4bb09fbe13113444d7510cc3ba7140bd1e557fd719c3ba08a5857eebbbcd9"} Oct 29 22:31:14.140196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:14.140169 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh" Oct 29 22:31:14.140358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:14.140186 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" event={"ID":"df4ab525-2e90-431a-b43c-0dfbea69b111","Type":"ContainerStarted","Data":"7b8f1dce522b5c4aac5f735616961bf7ae2613437cb05fc3ffe4887b3b94af49"} Oct 29 22:31:14.140358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:14.140248 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" event={"ID":"df4ab525-2e90-431a-b43c-0dfbea69b111","Type":"ContainerStarted","Data":"99994fc1aa721894c034b702cf506193966e82daf67e6e00dc5508acf75d5aa2"} Oct 29 22:31:14.140358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:14.140310 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:14.186317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:14.186270 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" podStartSLOduration=2.186257889 podStartE2EDuration="2.186257889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:14.184518113 +0000 UTC m=+2939.659106533" watchObservedRunningTime="2026-10-29 22:31:14.186257889 +0000 UTC m=+2939.660846307" Oct 29 22:31:14.249637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:14.249598 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh"] Oct 29 22:31:14.254339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:14.254309 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-jrgrh"] Oct 29 22:31:14.648196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:14.648164 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03ee8fc-3424-4232-85ad-96a512607f60" path="/var/lib/kubelet/pods/b03ee8fc-3424-4232-85ad-96a512607f60/volumes" Oct 29 22:31:15.154004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:15.153969 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" event={"ID":"81383129-aa94-4349-a1c2-3a3a9240be1b","Type":"ContainerStarted","Data":"5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f"} Oct 29 22:31:15.154473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:15.154068 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:15.202974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:15.202894 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" podStartSLOduration=2.202874348 podStartE2EDuration="2.202874348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:15.197823586 +0000 UTC m=+2940.672412004" watchObservedRunningTime="2026-10-29 22:31:15.202874348 +0000 UTC m=+2940.677462767" Oct 29 22:31:15.434485 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:15.431582 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:15.539170 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:31:15.539132 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:31:17.211445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:17.211249 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:18.808929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:18.807571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:19.764532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:19.764503 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:31:20.418931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:20.413870 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:22.119930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:22.116919 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:23.655926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:23.646617 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:25.158520 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:25.158488 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:25.309785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:25.309748 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:26.166202 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:26.166171 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:31:26.254786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:26.254743 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6"] Oct 29 22:31:26.934926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:26.934869 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:28.185891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:28.185841 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:28.363925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:28.359170 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:28.849939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:28.847818 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:29.340894 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:29.340859 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:29.519405 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:29.519364 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:29.993787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:29.992275 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:30.913850 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:30.913808 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:31.255736 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.255636 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" podUID="df4ab525-2e90-431a-b43c-0dfbea69b111" containerName="payload-processing" containerID="cri-o://7b8f1dce522b5c4aac5f735616961bf7ae2613437cb05fc3ffe4887b3b94af49" gracePeriod=25 Oct 29 22:31:31.399626 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.399594 4952 generic.go:356] "Generic (PLEG): container finished" podID="df4ab525-2e90-431a-b43c-0dfbea69b111" containerID="7b8f1dce522b5c4aac5f735616961bf7ae2613437cb05fc3ffe4887b3b94af49" exitCode=0 Oct 29 22:31:31.399790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.399680 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" event={"ID":"df4ab525-2e90-431a-b43c-0dfbea69b111","Type":"ContainerDied","Data":"7b8f1dce522b5c4aac5f735616961bf7ae2613437cb05fc3ffe4887b3b94af49"} Oct 29 22:31:31.568912 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.568879 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:31.659490 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.659452 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-routing-overlay-volume\") pod \"df4ab525-2e90-431a-b43c-0dfbea69b111\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " Oct 29 22:31:31.659678 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.659568 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-provider-credentials\") pod \"df4ab525-2e90-431a-b43c-0dfbea69b111\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " Oct 29 22:31:31.659884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.659859 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-kube-api-access-j9g9m\" (UniqueName: \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-kube-api-access-j9g9m\") pod \"df4ab525-2e90-431a-b43c-0dfbea69b111\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " Oct 29 22:31:31.660020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.659928 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-plugins-config-volume\") pod \"df4ab525-2e90-431a-b43c-0dfbea69b111\" (UID: \"df4ab525-2e90-431a-b43c-0dfbea69b111\") " Oct 29 22:31:31.660380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.660103 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-routing-overlay-volume" pod "df4ab525-2e90-431a-b43c-0dfbea69b111" (UID: "df4ab525-2e90-431a-b43c-0dfbea69b111"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:31.660589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.660562 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-plugins-config-volume" pod "df4ab525-2e90-431a-b43c-0dfbea69b111" (UID: "df4ab525-2e90-431a-b43c-0dfbea69b111"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:31:31.661107 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.661081 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:31.661311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.661292 4952 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df4ab525-2e90-431a-b43c-0dfbea69b111-routing-overlay-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:31.668954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.662867 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-provider-credentials" pod "df4ab525-2e90-431a-b43c-0dfbea69b111" (UID: "df4ab525-2e90-431a-b43c-0dfbea69b111"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:31.668954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.662897 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-kube-api-access-j9g9m" pod "df4ab525-2e90-431a-b43c-0dfbea69b111" (UID: "df4ab525-2e90-431a-b43c-0dfbea69b111"). InnerVolumeSpecName "kube-api-access-j9g9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:31:31.763767 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.763679 4952 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-provider-credentials\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:31.763767 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:31.763769 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9g9m\" (UniqueName: \"kubernetes.io/projected/df4ab525-2e90-431a-b43c-0dfbea69b111-kube-api-access-j9g9m\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:31:32.409602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:32.409569 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" Oct 29 22:31:32.410083 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:32.409562 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6" event={"ID":"df4ab525-2e90-431a-b43c-0dfbea69b111","Type":"ContainerDied","Data":"99994fc1aa721894c034b702cf506193966e82daf67e6e00dc5508acf75d5aa2"} Oct 29 22:31:32.410083 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:32.409711 4952 scope.go:122] "RemoveContainer" containerID="7b8f1dce522b5c4aac5f735616961bf7ae2613437cb05fc3ffe4887b3b94af49" Oct 29 22:31:32.471718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:32.471689 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6"] Oct 29 22:31:32.485091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:32.485046 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7cccp5mk6"] Oct 29 22:31:32.536924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:32.530614 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:32.649888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:32.649849 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4ab525-2e90-431a-b43c-0dfbea69b111" path="/var/lib/kubelet/pods/df4ab525-2e90-431a-b43c-0dfbea69b111/volumes" Oct 29 22:31:34.115151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:34.115115 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:35.718476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:35.715458 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:36.955535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:36.955496 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:37.127477 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:37.127422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:37.378134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:37.378091 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:39.023462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:39.023422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:40.699928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:40.699602 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:42.522560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:42.522516 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:44.026660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:44.026620 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:44.212021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:44.211980 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:44.450134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:44.447723 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:44.613499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:44.613455 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:45.045303 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:45.045262 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:45.263794 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:45.263757 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:45.610569 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:31:45.610529 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:31:46.505262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:46.505221 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:48.475927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:48.473214 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:48.655017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:48.654984 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:48.827621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:48.827585 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:50.207658 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:50.207617 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:50.402052 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:50.402011 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:50.587203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:50.587160 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:52.181170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:52.178416 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:53.815644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:53.815590 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:55.355642 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:55.355604 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:55.525577 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:55.525531 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:57.044930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:57.044872 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:58.715934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:31:58.715884 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:00.130832 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.130787 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs"] Oct 29 22:32:00.132104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.132079 4952 cpu_manager.go:414] "containerMap: removing container" podUID="df4ab525-2e90-431a-b43c-0dfbea69b111" containerName="payload-processing" Oct 29 22:32:00.132104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.132104 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4ab525-2e90-431a-b43c-0dfbea69b111" containerName="payload-processing" Oct 29 22:32:00.132379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.132360 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4ab525-2e90-431a-b43c-0dfbea69b111" containerName="payload-processing" Oct 29 22:32:00.136666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.136644 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.160530 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.160500 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs"] Oct 29 22:32:00.165806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.165780 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-90f46c-serving-cert\"" Oct 29 22:32:00.263350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.263309 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-metrics-certs\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.263517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.263358 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hz4k\" (UniqueName: \"kubernetes.io/projected/bc7020b1-ea9e-450d-936b-40965f599b95-kube-api-access-2hz4k\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.263517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.263473 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7020b1-ea9e-450d-936b-40965f599b95-openshift-service-ca\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.263640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.263625 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-maas-api-tls\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.267362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.267332 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:00.365382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.365347 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-metrics-certs\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.365590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.365398 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hz4k\" (UniqueName: \"kubernetes.io/projected/bc7020b1-ea9e-450d-936b-40965f599b95-kube-api-access-2hz4k\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.365590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.365469 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7020b1-ea9e-450d-936b-40965f599b95-openshift-service-ca\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.365712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.365650 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-maas-api-tls\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.367029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.366988 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7020b1-ea9e-450d-936b-40965f599b95-openshift-service-ca\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.369290 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.369263 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-maas-api-tls\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.398482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.397576 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-metrics-certs\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.411240 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.410642 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hz4k\" (UniqueName: \"kubernetes.io/projected/bc7020b1-ea9e-450d-936b-40965f599b95-kube-api-access-2hz4k\") pod \"maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.453471 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.453430 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:00.482300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.482268 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:00.868109 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:32:00.866749 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7020b1_ea9e_450d_936b_40965f599b95.slice/crio-f4d1d4a26ec6d037195b9e6521121d146ed51c142501d2b72b1cac8879c10625 WatchSource:0}: Error finding container f4d1d4a26ec6d037195b9e6521121d146ed51c142501d2b72b1cac8879c10625: Status 404 returned error can't find the container with id f4d1d4a26ec6d037195b9e6521121d146ed51c142501d2b72b1cac8879c10625 Oct 29 22:32:00.874919 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:00.874878 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs"] Oct 29 22:32:01.150862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:01.150767 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" event={"ID":"bc7020b1-ea9e-450d-936b-40965f599b95","Type":"ContainerStarted","Data":"f4d1d4a26ec6d037195b9e6521121d146ed51c142501d2b72b1cac8879c10625"} Oct 29 22:32:02.129848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:02.129801 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:02.184196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:02.184156 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" event={"ID":"bc7020b1-ea9e-450d-936b-40965f599b95","Type":"ContainerStarted","Data":"e9654c007c2c7c2bdf08e69be1f314d2354be3bd62cdceeab958c4116ace8757"} Oct 29 22:32:02.186011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:02.185984 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:02.247242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:02.247185 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" podStartSLOduration=1.641173165 podStartE2EDuration="2.247169426s" totalImagesPullingTime="605.996261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:02.24162517 +0000 UTC m=+2987.716213594" watchObservedRunningTime="2026-10-29 22:32:02.247169426 +0000 UTC m=+2987.721757894" Oct 29 22:32:03.704933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:03.703168 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:05.610409 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:05.610371 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:06.919329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:06.919286 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:08.529928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:08.527984 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:09.210462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:09.210429 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:32:09.474874 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:09.474778 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:09.672369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:09.672327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:10.789174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.789140 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq"] Oct 29 22:32:10.803928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.796291 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:10.828619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.828576 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:10.858690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.858657 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"routing-overlay\"" Oct 29 22:32:10.867583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.867553 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq"] Oct 29 22:32:10.875824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.875785 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z2v2\" (UniqueName: \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-kube-api-access-7z2v2\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:10.876055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.876032 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:10.876134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.876104 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:10.876201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.876155 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:10.876583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.876560 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"agc-cred-6cfe15a5e916\"" Oct 29 22:32:10.877028 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.876770 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"openshift-service-ca.crt\"" Oct 29 22:32:10.877160 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.876839 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-adfb9d\"" Oct 29 22:32:10.877246 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.876875 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"kube-root-ca.crt\"" Oct 29 22:32:10.877329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.876983 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-adfb9d\"/\"payload-processing-external-model-e2e--b64eb102-dockercfg-2z6nl\"" Oct 29 22:32:10.995928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.978512 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:10.996405 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.996371 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:10.996619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.996601 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:10.996889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:10.996872 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2v2\" (UniqueName: \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-kube-api-access-7z2v2\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:11.004250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.004217 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:11.053436 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.012274 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:11.054086 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.049736 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z2v2\" (UniqueName: \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-kube-api-access-7z2v2\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:11.092923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.081215 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:11.169927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.152853 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:11.229034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.228178 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:11.354242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.354201 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9"] Oct 29 22:32:11.361736 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.361695 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.383014 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.381229 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-26abe8-serving-cert\"" Oct 29 22:32:11.419930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.407378 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9"] Oct 29 22:32:11.455928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.442444 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgm8t\" (UniqueName: \"kubernetes.io/projected/d6b70588-0e21-4abb-bdec-e51dc4a54291-kube-api-access-zgm8t\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.455928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.442738 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b70588-0e21-4abb-bdec-e51dc4a54291-openshift-service-ca\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.455928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.442879 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-maas-api-tls\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.455928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.442999 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-metrics-certs\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.519943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.516628 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:11.552933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.552642 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-metrics-certs\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.552933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.552833 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgm8t\" (UniqueName: \"kubernetes.io/projected/d6b70588-0e21-4abb-bdec-e51dc4a54291-kube-api-access-zgm8t\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.553318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.553026 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b70588-0e21-4abb-bdec-e51dc4a54291-openshift-service-ca\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.553318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.553133 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-maas-api-tls\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.553441 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:32:11.553361 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-serving-cert: secret "maas-api-e2e-inf-b-26abe8-serving-cert" not found Oct 29 22:32:11.553497 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:32:11.553444 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-maas-api-tls podName:d6b70588-0e21-4abb-bdec-e51dc4a54291 nodeName:}" failed. No retries permitted until 2026-10-29 22:32:12.053422873 +0000 UTC m=+2997.528011282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-maas-api-tls") pod "maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" (UID: "d6b70588-0e21-4abb-bdec-e51dc4a54291") : secret "maas-api-e2e-inf-b-26abe8-serving-cert" not found Oct 29 22:32:11.558928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.554647 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b70588-0e21-4abb-bdec-e51dc4a54291-openshift-service-ca\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.713929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.711051 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:11.902831 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:32:11.898335 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f80bd7e_b83a_4769_9d8a_4b8c15d942c1.slice/crio-2f35ff5c61f2222232512a3ce5f15e581e2cb07bfedb810f10b637cf109cf1d3 WatchSource:0}: Error finding container 2f35ff5c61f2222232512a3ce5f15e581e2cb07bfedb810f10b637cf109cf1d3: Status 404 returned error can't find the container with id 2f35ff5c61f2222232512a3ce5f15e581e2cb07bfedb810f10b637cf109cf1d3 Oct 29 22:32:11.985723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.978172 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:11.985723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.979514 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-metrics-certs\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:11.985723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:11.985625 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgm8t\" (UniqueName: \"kubernetes.io/projected/d6b70588-0e21-4abb-bdec-e51dc4a54291-kube-api-access-zgm8t\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:12.013932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.003787 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq"] Oct 29 22:32:12.118009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.107747 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-maas-api-tls\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:12.135937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.135855 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-maas-api-tls\") pod \"maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:12.290281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.287433 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:12.303391 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.303359 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:12.469970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.463521 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:12.469970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.469302 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" event={"ID":"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1","Type":"ContainerStarted","Data":"e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230"} Oct 29 22:32:12.469970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.469372 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" event={"ID":"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1","Type":"ContainerStarted","Data":"2f35ff5c61f2222232512a3ce5f15e581e2cb07bfedb810f10b637cf109cf1d3"} Oct 29 22:32:12.470693 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.470665 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:12.517724 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.517668 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" podStartSLOduration=2.517651328 podStartE2EDuration="2.517651328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:12.515947576 +0000 UTC m=+2997.990535995" watchObservedRunningTime="2026-10-29 22:32:12.517651328 +0000 UTC m=+2997.992239746" Oct 29 22:32:12.697051 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:32:12.697019 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b70588_0e21_4abb_bdec_e51dc4a54291.slice/crio-3a1313300b682b53cf5d4c82aa433cfcad4cd2f3cbc31e14789023ebe71fc275 WatchSource:0}: Error finding container 3a1313300b682b53cf5d4c82aa433cfcad4cd2f3cbc31e14789023ebe71fc275: Status 404 returned error can't find the container with id 3a1313300b682b53cf5d4c82aa433cfcad4cd2f3cbc31e14789023ebe71fc275 Oct 29 22:32:12.697420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:12.697397 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9"] Oct 29 22:32:13.538546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:13.538449 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" event={"ID":"d6b70588-0e21-4abb-bdec-e51dc4a54291","Type":"ContainerStarted","Data":"1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce"} Oct 29 22:32:13.538546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:13.538529 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" event={"ID":"d6b70588-0e21-4abb-bdec-e51dc4a54291","Type":"ContainerStarted","Data":"3a1313300b682b53cf5d4c82aa433cfcad4cd2f3cbc31e14789023ebe71fc275"} Oct 29 22:32:13.539007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:13.538578 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:13.592972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:13.592897 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" podStartSLOduration=2.207102221 podStartE2EDuration="2.592882506s" totalImagesPullingTime="385.780285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:13.587771766 +0000 UTC m=+2999.062360187" watchObservedRunningTime="2026-10-29 22:32:13.592882506 +0000 UTC m=+2999.067470924" Oct 29 22:32:14.173927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:14.165600 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:15.530603 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:32:15.530559 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:32:15.770614 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:15.770567 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:32:15.815130 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:15.808553 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:16.111705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:16.111662 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:32:16.132680 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:16.132649 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:32:17.980686 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:17.980120 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:19.586927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:19.586157 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:32:19.603964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:19.603296 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:21.328938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:21.322113 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:22.947956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:22.944599 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:22.982856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:22.982471 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp"] Oct 29 22:32:24.569538 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:24.569499 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:24.582167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:24.582133 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:26.210847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:26.208856 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:27.916556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:27.916450 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:27.983500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:27.983437 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" podUID="c9929e56-05ff-4138-96af-66fd1d7b329f" containerName="payload-processing" containerID="cri-o://92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211" gracePeriod=25 Oct 29 22:32:28.292987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.292959 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:32:28.362581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.362545 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-routing-overlay-volume\") pod \"c9929e56-05ff-4138-96af-66fd1d7b329f\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " Oct 29 22:32:28.362785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.362697 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-provider-credentials\") pod \"c9929e56-05ff-4138-96af-66fd1d7b329f\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " Oct 29 22:32:28.362900 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.362881 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-kube-api-access-77rqb\" (UniqueName: \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-kube-api-access-77rqb\") pod \"c9929e56-05ff-4138-96af-66fd1d7b329f\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " Oct 29 22:32:28.362981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.362940 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-routing-overlay-volume" pod "c9929e56-05ff-4138-96af-66fd1d7b329f" (UID: "c9929e56-05ff-4138-96af-66fd1d7b329f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:28.363070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.363041 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-plugins-config-volume\") pod \"c9929e56-05ff-4138-96af-66fd1d7b329f\" (UID: \"c9929e56-05ff-4138-96af-66fd1d7b329f\") " Oct 29 22:32:28.363983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.363953 4952 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-routing-overlay-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:32:28.364305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.364277 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-plugins-config-volume" pod "c9929e56-05ff-4138-96af-66fd1d7b329f" (UID: "c9929e56-05ff-4138-96af-66fd1d7b329f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:28.365620 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.365593 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-provider-credentials" pod "c9929e56-05ff-4138-96af-66fd1d7b329f" (UID: "c9929e56-05ff-4138-96af-66fd1d7b329f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:28.370296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.370272 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-kube-api-access-77rqb" pod "c9929e56-05ff-4138-96af-66fd1d7b329f" (UID: "c9929e56-05ff-4138-96af-66fd1d7b329f"). InnerVolumeSpecName "kube-api-access-77rqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:28.465726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.465686 4952 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-provider-credentials\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:32:28.465726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.465723 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77rqb\" (UniqueName: \"kubernetes.io/projected/c9929e56-05ff-4138-96af-66fd1d7b329f-kube-api-access-77rqb\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:32:28.465726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.465732 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9929e56-05ff-4138-96af-66fd1d7b329f-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:32:28.752664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.752626 4952 generic.go:356] "Generic (PLEG): container finished" podID="c9929e56-05ff-4138-96af-66fd1d7b329f" containerID="92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211" exitCode=0 Oct 29 22:32:28.752664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.752671 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" Oct 29 22:32:28.752954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.752689 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" event={"ID":"c9929e56-05ff-4138-96af-66fd1d7b329f","Type":"ContainerDied","Data":"92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211"} Oct 29 22:32:28.752954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.752731 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp" event={"ID":"c9929e56-05ff-4138-96af-66fd1d7b329f","Type":"ContainerDied","Data":"7f7da758dcf034b582d3b7034d02d28601eda4e1af86115a0019db7052ac854d"} Oct 29 22:32:28.752954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.752753 4952 scope.go:122] "RemoveContainer" containerID="92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211" Oct 29 22:32:28.768120 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.768094 4952 scope.go:122] "RemoveContainer" containerID="92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211" Oct 29 22:32:28.768445 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:32:28.768418 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211\": container with ID starting with 92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211 not found: ID does not exist" containerID="92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211" Oct 29 22:32:28.768539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.768453 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211"} err="failed to get container status \"92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211\": rpc error: code = NotFound desc = could not find container \"92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211\": container with ID starting with 92a3619055456e28ba1610ef8fa328b281a81c9967f9c32003ab080936406211 not found: ID does not exist" Oct 29 22:32:28.813285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.813253 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp"] Oct 29 22:32:28.822457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:28.822427 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b9ed94/payload-processing-external-model-e2e-extmod-a-b9ed94-66c52vvpp"] Oct 29 22:32:29.533931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:29.531249 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:30.648840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:30.648800 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9929e56-05ff-4138-96af-66fd1d7b329f" path="/var/lib/kubelet/pods/c9929e56-05ff-4138-96af-66fd1d7b329f/volumes" Oct 29 22:32:31.136026 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:31.133182 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:32.561023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:32.560983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:32.767944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:32.767888 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:33.070686 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:33.070644 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:33.558953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:33.558350 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:33.804579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:33.804539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:33.838819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:33.838771 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 29 22:32:34.317748 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:34.317658 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:34.548129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:34.530370 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:34.978407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:34.977858 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq"] Oct 29 22:32:35.058979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:35.058383 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:35.084935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:35.084630 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.091859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:35.091817 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.268101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:35.268006 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:36.511287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:36.511250 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:36.825975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:36.825940 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:37.009468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:37.009412 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:38.540089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:38.540051 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:38.718191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:38.718153 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:39.103047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:39.102962 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:39.415401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:39.415313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:39.978967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:39.978894 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" podUID="3f80bd7e-b83a-4769-9d8a-4b8c15d942c1" containerName="payload-processing" containerID="cri-o://e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230" gracePeriod=25 Oct 29 22:32:40.297543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.297517 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:40.357973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.357927 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-plugins-config-volume\") pod \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " Oct 29 22:32:40.358197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.357988 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-routing-overlay-volume\") pod \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " Oct 29 22:32:40.358510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.358277 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-provider-credentials\") pod \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " Oct 29 22:32:40.358849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.358651 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-routing-overlay-volume" pod "3f80bd7e-b83a-4769-9d8a-4b8c15d942c1" (UID: "3f80bd7e-b83a-4769-9d8a-4b8c15d942c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:40.358849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.358708 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-kube-api-access-7z2v2\" (UniqueName: \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-kube-api-access-7z2v2\") pod \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\" (UID: \"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1\") " Oct 29 22:32:40.358849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.358806 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-plugins-config-volume" pod "3f80bd7e-b83a-4769-9d8a-4b8c15d942c1" (UID: "3f80bd7e-b83a-4769-9d8a-4b8c15d942c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:32:40.359511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.359482 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:32:40.359615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.359514 4952 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-routing-overlay-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:32:40.362363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.362330 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-provider-credentials" pod "3f80bd7e-b83a-4769-9d8a-4b8c15d942c1" (UID: "3f80bd7e-b83a-4769-9d8a-4b8c15d942c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:40.362467 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.362390 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-kube-api-access-7z2v2" pod "3f80bd7e-b83a-4769-9d8a-4b8c15d942c1" (UID: "3f80bd7e-b83a-4769-9d8a-4b8c15d942c1"). InnerVolumeSpecName "kube-api-access-7z2v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:32:40.444554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.444510 4952 generic.go:356] "Generic (PLEG): container finished" podID="3f80bd7e-b83a-4769-9d8a-4b8c15d942c1" containerID="e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230" exitCode=0 Oct 29 22:32:40.444760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.444571 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" Oct 29 22:32:40.444760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.444594 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" event={"ID":"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1","Type":"ContainerDied","Data":"e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230"} Oct 29 22:32:40.444760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.444662 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq" event={"ID":"3f80bd7e-b83a-4769-9d8a-4b8c15d942c1","Type":"ContainerDied","Data":"2f35ff5c61f2222232512a3ce5f15e581e2cb07bfedb810f10b637cf109cf1d3"} Oct 29 22:32:40.444760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.444695 4952 scope.go:122] "RemoveContainer" containerID="e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230" Oct 29 22:32:40.461559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.461531 4952 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-provider-credentials\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:32:40.461559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.461559 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z2v2\" (UniqueName: \"kubernetes.io/projected/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1-kube-api-access-7z2v2\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:32:40.465987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.465967 4952 scope.go:122] "RemoveContainer" containerID="e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230" Oct 29 22:32:40.466270 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:32:40.466251 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230\": container with ID starting with e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230 not found: ID does not exist" containerID="e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230" Oct 29 22:32:40.466334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.466277 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230"} err="failed to get container status \"e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230\": rpc error: code = NotFound desc = could not find container \"e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230\": container with ID starting with e39a191a2f740d6fa40008489d6422f905eb169f6a52cf0275aa175fd477c230 not found: ID does not exist" Oct 29 22:32:40.509915 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.509871 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq"] Oct 29 22:32:40.521247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.521164 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-adfb9d/payload-processing-external-model-e2e-extmod-b-adfb9d-749fmrdnq"] Oct 29 22:32:40.650210 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.650173 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f80bd7e-b83a-4769-9d8a-4b8c15d942c1" path="/var/lib/kubelet/pods/3f80bd7e-b83a-4769-9d8a-4b8c15d942c1/volumes" Oct 29 22:32:40.651395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:40.651364 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:41.230929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:41.228511 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:41.638932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:41.636544 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:41.813435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:41.813394 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:42.121833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:42.121796 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:42.489707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:42.489621 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:42.771990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:42.771882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:42.979772 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:42.979693 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:43.186537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:43.183790 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:44.794054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:44.794017 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:45.529803 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:32:45.529761 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:32:46.407746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:46.407701 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:46.612630 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:46.612588 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:46.813563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:46.813469 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:47.186146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:47.186110 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:47.678889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:47.678804 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:47.887926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:47.880415 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:48.234231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:48.234118 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:48.643947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:48.642515 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:48.864740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:48.864700 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:50.607796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:50.607759 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:50.814482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:50.814445 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:51.140499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.140462 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:51.499679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.499561 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:51.729365 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.727359 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:51.823139 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.823040 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh"] Oct 29 22:32:51.824623 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.824589 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c9929e56-05ff-4138-96af-66fd1d7b329f" containerName="payload-processing" Oct 29 22:32:51.825247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.825226 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9929e56-05ff-4138-96af-66fd1d7b329f" containerName="payload-processing" Oct 29 22:32:51.825428 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.825414 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3f80bd7e-b83a-4769-9d8a-4b8c15d942c1" containerName="payload-processing" Oct 29 22:32:51.825521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.825510 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f80bd7e-b83a-4769-9d8a-4b8c15d942c1" containerName="payload-processing" Oct 29 22:32:51.829224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.829197 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9929e56-05ff-4138-96af-66fd1d7b329f" containerName="payload-processing" Oct 29 22:32:51.829404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.829391 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f80bd7e-b83a-4769-9d8a-4b8c15d942c1" containerName="payload-processing" Oct 29 22:32:51.898298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:51.898263 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.059269 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.048332 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-52429a-openshift-default-dockercfg-rbvt2\"" Oct 29 22:32:52.059269 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.050614 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh"] Oct 29 22:32:52.066454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.066408 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.075963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.075058 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.075963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.075136 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dd3058-c239-41cc-88dc-5d492be03792-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.075963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.075287 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.075963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.075315 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.075963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.075366 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dd3058-c239-41cc-88dc-5d492be03792-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.075963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.075461 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dbs\" (UniqueName: \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-kube-api-access-s9dbs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.075963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.075531 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.075963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.075561 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.146394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.144659 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:52.257302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.257236 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.257518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.257340 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dd3058-c239-41cc-88dc-5d492be03792-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.257518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.257509 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.257662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.257541 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.257662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.257590 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dd3058-c239-41cc-88dc-5d492be03792-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.257775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.257697 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dbs\" (UniqueName: \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-kube-api-access-s9dbs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.257775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.257764 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.257890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.257801 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.258035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.258001 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.260000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.259939 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.260453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.260418 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.262220 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.262194 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.262849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.262824 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.263739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.263719 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dd3058-c239-41cc-88dc-5d492be03792-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.321552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.313082 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.321552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.314443 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dd3058-c239-41cc-88dc-5d492be03792-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.354284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.353673 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:52.361175 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.360510 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dbs\" (UniqueName: \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-kube-api-access-s9dbs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.369570 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.369501 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.407704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.407659 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:52.768700 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:32:52.768656 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dd3058_c239_41cc_88dc_5d492be03792.slice/crio-2e115deea538c44ed63c194d2ae2bf9e8975bc8d863e830fee1ff0e3b0fad2a4 WatchSource:0}: Error finding container 2e115deea538c44ed63c194d2ae2bf9e8975bc8d863e830fee1ff0e3b0fad2a4: Status 404 returned error can't find the container with id 2e115deea538c44ed63c194d2ae2bf9e8975bc8d863e830fee1ff0e3b0fad2a4 Oct 29 22:32:52.770241 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.770212 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh"] Oct 29 22:32:52.772483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.772445 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:32:52.772608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.772537 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:32:52.772608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:52.772588 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:32:53.153930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:53.153819 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:53.247590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:53.247545 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" event={"ID":"e7dd3058-c239-41cc-88dc-5d492be03792","Type":"ContainerStarted","Data":"8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a"} Oct 29 22:32:53.247811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:53.247629 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" event={"ID":"e7dd3058-c239-41cc-88dc-5d492be03792","Type":"ContainerStarted","Data":"2e115deea538c44ed63c194d2ae2bf9e8975bc8d863e830fee1ff0e3b0fad2a4"} Oct 29 22:32:53.331982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:53.328023 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" podStartSLOduration=2.3280064830000002 podStartE2EDuration="2.328006483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:53.321946178 +0000 UTC m=+3038.796534596" watchObservedRunningTime="2026-10-29 22:32:53.328006483 +0000 UTC m=+3038.802594900" Oct 29 22:32:53.373370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:53.373337 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:53.408634 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:53.408545 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:53.410114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:53.410084 4952 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:32:53.410222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:53.410146 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" podUID="e7dd3058-c239-41cc-88dc-5d492be03792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 29 22:32:53.775977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:53.767760 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:54.415167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:54.415129 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:55.294083 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:55.294028 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:55.295951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:55.295920 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:55.297898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:55.297875 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:32:55.909377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:55.909344 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:56.082434 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:56.082391 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:57.387928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:57.384568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:57.910969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:57.910884 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:58.119937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:58.113362 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:59.174946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:59.174138 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:59.218246 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:59.213284 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:59.353929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:59.349028 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:59.571760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:32:59.564130 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:00.174924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:00.174872 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:01.935342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:01.935301 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk"] Oct 29 22:33:01.941735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:01.941711 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:01.964541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:01.964508 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk"] Oct 29 22:33:01.969743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:01.969700 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d09411-serving-cert\"" Oct 29 22:33:02.017152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.017108 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf522b0-f6c8-40c3-8775-50ff08781fc1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.017463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.017303 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-maas-api-tls\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.017463 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.017363 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d6dt\" (UniqueName: \"kubernetes.io/projected/2cf522b0-f6c8-40c3-8775-50ff08781fc1-kube-api-access-2d6dt\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.017590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.017466 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-metrics-certs\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.067899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.067857 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:02.126926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.120222 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf522b0-f6c8-40c3-8775-50ff08781fc1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.126926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.120429 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-maas-api-tls\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.126926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.120489 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d6dt\" (UniqueName: \"kubernetes.io/projected/2cf522b0-f6c8-40c3-8775-50ff08781fc1-kube-api-access-2d6dt\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.126926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.120598 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-metrics-certs\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.156936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.141723 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-metrics-certs\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.157136 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:33:02.142301 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-serving-cert: secret "maas-api-e2e-inf-a-d09411-serving-cert" not found Oct 29 22:33:02.157203 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:33:02.157161 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-maas-api-tls podName:2cf522b0-f6c8-40c3-8775-50ff08781fc1 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:02.657130112 +0000 UTC m=+3048.131718518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-maas-api-tls") pod "maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" (UID: "2cf522b0-f6c8-40c3-8775-50ff08781fc1") : secret "maas-api-e2e-inf-a-d09411-serving-cert" not found Oct 29 22:33:02.174008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.143016 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf522b0-f6c8-40c3-8775-50ff08781fc1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.183000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.182963 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d6dt\" (UniqueName: \"kubernetes.io/projected/2cf522b0-f6c8-40c3-8775-50ff08781fc1-kube-api-access-2d6dt\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.238100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.237960 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:02.426998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.426957 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:02.735763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.735722 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-maas-api-tls\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.738668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.738643 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-maas-api-tls\") pod \"maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:02.862647 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:02.862603 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:03.468037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:03.467455 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk"] Oct 29 22:33:03.995305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:03.995254 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:04.096642 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:04.096041 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" event={"ID":"2cf522b0-f6c8-40c3-8775-50ff08781fc1","Type":"ContainerStarted","Data":"a9f1aec62f5d8fb6d9092a18044100ee79b8b513c6b8fc637ee513f6fc98984c"} Oct 29 22:33:05.106323 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:05.106284 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" event={"ID":"2cf522b0-f6c8-40c3-8775-50ff08781fc1","Type":"ContainerStarted","Data":"dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79"} Oct 29 22:33:05.106737 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:05.106348 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:05.157941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:05.157874 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" podStartSLOduration=3.578264388 podStartE2EDuration="4.157854102s" totalImagesPullingTime="579.589714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:05.152576714 +0000 UTC m=+3050.627165133" watchObservedRunningTime="2026-10-29 22:33:05.157854102 +0000 UTC m=+3050.632442523" Oct 29 22:33:05.844559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:05.843995 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:07.464343 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:07.464306 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:09.157115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:09.156948 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:10.371272 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:10.371231 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn"] Oct 29 22:33:10.371870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:10.371484 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" containerID="cri-o://bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71" gracePeriod=30 Oct 29 22:33:10.850930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:10.841316 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:11.125502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:11.125421 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:33:12.227320 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:12.227281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:12.477935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:12.477809 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:14.063056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:14.063007 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:14.113196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:14.113163 4952 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:33:14.113454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:14.113419 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:33:15.535648 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:33:15.535611 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:33:15.727365 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.727336 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:33:15.813109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.812770 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch"] Oct 29 22:33:15.863928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.850109 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.877523 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c932aaf0-9272-4567-aaaa-90a392720191-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c932aaf0-9272-4567-aaaa-90a392720191-istio-podinfo\") pod \"c932aaf0-9272-4567-aaaa-90a392720191\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.877613 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-envoy\") pod \"c932aaf0-9272-4567-aaaa-90a392720191\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.877686 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-kube-api-access-vpphr\" (UniqueName: \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-kube-api-access-vpphr\") pod \"c932aaf0-9272-4567-aaaa-90a392720191\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.877730 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-certs\") pod \"c932aaf0-9272-4567-aaaa-90a392720191\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.877918 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-socket\") pod \"c932aaf0-9272-4567-aaaa-90a392720191\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.877982 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-istio-token\" (UniqueName: \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-istio-token\") pod \"c932aaf0-9272-4567-aaaa-90a392720191\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.878165 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-data\") pod \"c932aaf0-9272-4567-aaaa-90a392720191\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.878222 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-credential-socket\") pod \"c932aaf0-9272-4567-aaaa-90a392720191\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.878332 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c932aaf0-9272-4567-aaaa-90a392720191-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c932aaf0-9272-4567-aaaa-90a392720191-istiod-ca-cert\") pod \"c932aaf0-9272-4567-aaaa-90a392720191\" (UID: \"c932aaf0-9272-4567-aaaa-90a392720191\") " Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.884967 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-certs" pod "c932aaf0-9272-4567-aaaa-90a392720191" (UID: "c932aaf0-9272-4567-aaaa-90a392720191"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:33:15.895552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.893455 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-socket" pod "c932aaf0-9272-4567-aaaa-90a392720191" (UID: "c932aaf0-9272-4567-aaaa-90a392720191"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:33:15.913210 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.900993 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-data" pod "c932aaf0-9272-4567-aaaa-90a392720191" (UID: "c932aaf0-9272-4567-aaaa-90a392720191"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:33:15.919080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.913964 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-istio-token" pod "c932aaf0-9272-4567-aaaa-90a392720191" (UID: "c932aaf0-9272-4567-aaaa-90a392720191"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:15.919080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.915247 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-credential-socket" pod "c932aaf0-9272-4567-aaaa-90a392720191" (UID: "c932aaf0-9272-4567-aaaa-90a392720191"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:33:15.942009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.926171 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c932aaf0-9272-4567-aaaa-90a392720191-istiod-ca-cert" pod "c932aaf0-9272-4567-aaaa-90a392720191" (UID: "c932aaf0-9272-4567-aaaa-90a392720191"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:33:15.942009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.926305 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-envoy" pod "c932aaf0-9272-4567-aaaa-90a392720191" (UID: "c932aaf0-9272-4567-aaaa-90a392720191"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:33:15.966037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.957079 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:33:15.966037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.957120 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:33:15.966037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.957136 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c932aaf0-9272-4567-aaaa-90a392720191-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:33:15.966037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.957157 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:33:15.966037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.957171 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:33:15.966037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.957184 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c932aaf0-9272-4567-aaaa-90a392720191-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:33:15.966037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.957199 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:33:15.966037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.957927 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" Oct 29 22:33:15.966037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.957957 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" Oct 29 22:33:15.966037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.958401 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c932aaf0-9272-4567-aaaa-90a392720191" containerName="istio-proxy" Oct 29 22:33:15.966701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.966589 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:15.969106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.969071 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c932aaf0-9272-4567-aaaa-90a392720191-istio-podinfo" pod "c932aaf0-9272-4567-aaaa-90a392720191" (UID: "c932aaf0-9272-4567-aaaa-90a392720191"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:33:15.971981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.971883 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-kube-api-access-vpphr" pod "c932aaf0-9272-4567-aaaa-90a392720191" (UID: "c932aaf0-9272-4567-aaaa-90a392720191"). InnerVolumeSpecName "kube-api-access-vpphr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:33:15.974842 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:15.974810 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch"] Oct 29 22:33:16.058514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.058477 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-credential-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.058726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.058651 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f78edf-e3c7-4336-b114-312d2b342149-istiod-ca-cert\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.058726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.058692 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f78edf-e3c7-4336-b114-312d2b342149-istio-podinfo\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.058856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.058820 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-istio-token\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.058931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.058855 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-data\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.058997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.058974 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpkj\" (UniqueName: \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-kube-api-access-tjpkj\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.059055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.059002 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.059055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.059045 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-envoy\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.059170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.059147 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-certs\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.059338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.059318 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c932aaf0-9272-4567-aaaa-90a392720191-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:33:16.059430 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.059342 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpphr\" (UniqueName: \"kubernetes.io/projected/c932aaf0-9272-4567-aaaa-90a392720191-kube-api-access-vpphr\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:33:16.160373 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.160336 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-data\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.160562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.160444 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpkj\" (UniqueName: \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-kube-api-access-tjpkj\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.160562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.160478 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.160562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.160522 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-envoy\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.160562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.160553 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-certs\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.160775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.160604 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-credential-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.160947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.160820 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-data\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.161090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.161064 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.161164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.161067 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f78edf-e3c7-4336-b114-312d2b342149-istiod-ca-cert\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.161164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.161120 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-credential-socket\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.161164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.161003 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-certs\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.161323 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.161170 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f78edf-e3c7-4336-b114-312d2b342149-istio-podinfo\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.161323 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.161272 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-istio-token\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.161757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.161730 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f78edf-e3c7-4336-b114-312d2b342149-istiod-ca-cert\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.163279 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.163245 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-envoy\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.163713 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.163684 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f78edf-e3c7-4336-b114-312d2b342149-istio-podinfo\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.186268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.186243 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-istio-token\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.186414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.186394 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpkj\" (UniqueName: \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-kube-api-access-tjpkj\") pod \"e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.237015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.236880 4952 generic.go:356] "Generic (PLEG): container finished" podID="c932aaf0-9272-4567-aaaa-90a392720191" containerID="bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71" exitCode=0 Oct 29 22:33:16.237015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.236958 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" event={"ID":"c932aaf0-9272-4567-aaaa-90a392720191","Type":"ContainerDied","Data":"bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71"} Oct 29 22:33:16.237258 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.237030 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" event={"ID":"c932aaf0-9272-4567-aaaa-90a392720191","Type":"ContainerDied","Data":"2789279d4e6c2c3a78a0606d0f4d49b0125fe967758ffec18a39bcd6a2b5d21f"} Oct 29 22:33:16.237258 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.236971 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn" Oct 29 22:33:16.237258 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.237059 4952 scope.go:122] "RemoveContainer" containerID="bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71" Oct 29 22:33:16.251865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.251844 4952 scope.go:122] "RemoveContainer" containerID="bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71" Oct 29 22:33:16.252215 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:33:16.252188 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71\": container with ID starting with bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71 not found: ID does not exist" containerID="bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71" Oct 29 22:33:16.252280 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.252227 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71"} err="failed to get container status \"bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71\": rpc error: code = NotFound desc = could not find container \"bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71\": container with ID starting with bde8dc9f930786cefa37e8c75be9a8c373e52d7a28d0b8a930b26f08460f3b71 not found: ID does not exist" Oct 29 22:33:16.288764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.288736 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:16.298804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.298771 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn"] Oct 29 22:33:16.306319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.306291 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-ctgjn"] Oct 29 22:33:16.629891 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:33:16.629858 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f78edf_e3c7_4336_b114_312d2b342149.slice/crio-82dc41548797b776327d944d91893e6320eadab4d89a501a733ca3fefa28154e WatchSource:0}: Error finding container 82dc41548797b776327d944d91893e6320eadab4d89a501a733ca3fefa28154e: Status 404 returned error can't find the container with id 82dc41548797b776327d944d91893e6320eadab4d89a501a733ca3fefa28154e Oct 29 22:33:16.630420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.630397 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch"] Oct 29 22:33:16.633824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.633790 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:33:16.633929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.633864 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:33:16.633929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.633892 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:33:16.658504 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:16.658466 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c932aaf0-9272-4567-aaaa-90a392720191" path="/var/lib/kubelet/pods/c932aaf0-9272-4567-aaaa-90a392720191/volumes" Oct 29 22:33:17.319744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:17.319702 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" event={"ID":"95f78edf-e3c7-4336-b114-312d2b342149","Type":"ContainerStarted","Data":"e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d"} Oct 29 22:33:17.320055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:17.320030 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" event={"ID":"95f78edf-e3c7-4336-b114-312d2b342149","Type":"ContainerStarted","Data":"82dc41548797b776327d944d91893e6320eadab4d89a501a733ca3fefa28154e"} Oct 29 22:33:17.444444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:17.444383 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" podStartSLOduration=7.444361463 podStartE2EDuration="7.444361463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:17.401441369 +0000 UTC m=+3062.876029787" watchObservedRunningTime="2026-10-29 22:33:17.444361463 +0000 UTC m=+3062.918949881" Oct 29 22:33:17.711930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:17.711698 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:18.289234 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:18.289196 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:18.290847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:18.290820 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 29 22:33:18.290968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:18.290873 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 29 22:33:18.893931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:18.886041 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:19.289999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:19.289431 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 29 22:33:19.289999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:19.289496 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 29 22:33:19.744043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:19.737497 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:20.292395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:20.292353 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 29 22:33:20.293360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:20.292937 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 29 22:33:21.290928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:21.290496 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 29 22:33:21.290928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:21.290561 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 29 22:33:21.812928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:21.805484 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:22.209008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.208964 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt"] Oct 29 22:33:22.256353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.256316 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt"] Oct 29 22:33:22.272382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.256665 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.295929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.294297 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c06a2c\"/\"default-dockercfg-c5xsx\"" Oct 29 22:33:22.298140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.297235 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 29 22:33:22.298140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.297300 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 29 22:33:22.299153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.298993 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:22.308578 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.308547 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c06a2c\"/\"openshift-service-ca.crt\"" Oct 29 22:33:22.310109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.309132 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c06a2c\"/\"test-model-c06a2c-kserve-self-signed-certs\"" Oct 29 22:33:22.310297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.309233 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c06a2c\"/\"kube-root-ca.crt\"" Oct 29 22:33:22.328276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.326173 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf35311-cf34-4be6-847c-4088a59bf188-tls-certs\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.328276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.326305 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-model-cache\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.328276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.326497 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-home\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.328276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.326635 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s7zf\" (UniqueName: \"kubernetes.io/projected/9cf35311-cf34-4be6-847c-4088a59bf188-kube-api-access-7s7zf\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.328276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.326692 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-tmp-dir\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.328276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.326735 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-dshm\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.490808 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.456146 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-model-cache\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.633242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.622917 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-home\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.661926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.649438 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s7zf\" (UniqueName: \"kubernetes.io/projected/9cf35311-cf34-4be6-847c-4088a59bf188-kube-api-access-7s7zf\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.671028 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.662392 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-tmp-dir\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.695327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.695279 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-dshm\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.696763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.695712 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf35311-cf34-4be6-847c-4088a59bf188-tls-certs\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.714517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.663710 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-tmp-dir\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.729833 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.459579 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-model-cache\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.730077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.636360 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-home\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.814983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.814888 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf35311-cf34-4be6-847c-4088a59bf188-tls-certs\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.887115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.887079 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-dshm\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:22.914795 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:22.914706 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s7zf\" (UniqueName: \"kubernetes.io/projected/9cf35311-cf34-4be6-847c-4088a59bf188-kube-api-access-7s7zf\") pod \"test-model-c06a2c-kserve-6c9f47f669-5t8zt\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:23.129236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:23.129199 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:23.293333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:23.293286 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 29 22:33:23.293531 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:23.293345 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 29 22:33:23.593237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:23.590681 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt"] Oct 29 22:33:23.596741 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:33:23.596619 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cf35311_cf34_4be6_847c_4088a59bf188.slice/crio-ae3f442bd206ea47c34b65bfb8ddf8bf591c9fc23580bbd6f1d76726c8b937be WatchSource:0}: Error finding container ae3f442bd206ea47c34b65bfb8ddf8bf591c9fc23580bbd6f1d76726c8b937be: Status 404 returned error can't find the container with id ae3f442bd206ea47c34b65bfb8ddf8bf591c9fc23580bbd6f1d76726c8b937be Oct 29 22:33:24.026926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:24.021824 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:24.289730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:24.289688 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 29 22:33:24.289972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:24.289759 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 29 22:33:24.622121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:24.622070 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" event={"ID":"9cf35311-cf34-4be6-847c-4088a59bf188","Type":"ContainerStarted","Data":"5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9"} Oct 29 22:33:24.622121 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:24.622124 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" event={"ID":"9cf35311-cf34-4be6-847c-4088a59bf188","Type":"ContainerStarted","Data":"ae3f442bd206ea47c34b65bfb8ddf8bf591c9fc23580bbd6f1d76726c8b937be"} Oct 29 22:33:24.671139 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:24.671075 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" podStartSLOduration=2.67105767 podStartE2EDuration="2.67105767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:24.669158871 +0000 UTC m=+3070.143747291" watchObservedRunningTime="2026-10-29 22:33:24.67105767 +0000 UTC m=+3070.145646088" Oct 29 22:33:25.130759 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:25.130722 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:25.295333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:25.295300 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:25.295826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:25.295797 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:25.296571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:25.296547 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:33:25.626084 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:25.626043 4952 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:25.626287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:25.626110 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:25.841134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:25.841007 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:27.526547 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:27.526506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:28.664557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:28.664466 4952 generic.go:356] "Generic (PLEG): container finished" podID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerID="52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21" exitCode=0 Oct 29 22:33:28.664557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:28.664539 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" event={"ID":"d84b5278-d243-470c-abf8-e6e8e9aa3e62","Type":"ContainerDied","Data":"52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21"} Oct 29 22:33:28.665264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:28.665083 4952 scope.go:122] "RemoveContainer" containerID="52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21" Oct 29 22:33:28.667252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:28.667219 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:33:28.667374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:28.667304 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:33:28.667374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:28.667352 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:33:29.333581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:29.333431 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:29.680891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:29.680806 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" event={"ID":"d84b5278-d243-470c-abf8-e6e8e9aa3e62","Type":"ContainerStarted","Data":"2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b"} Oct 29 22:33:30.633936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:30.631274 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:33:30.633936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:30.631411 4952 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:33:30.633936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:30.631465 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 22:33:31.525935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:31.518561 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:31.636929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:31.636318 4952 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:33:31.636929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:31.636399 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 22:33:32.636930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:32.635345 4952 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:33:32.636930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:32.635410 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 22:33:33.141045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:33.141001 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:33.198399 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:33.198363 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:33.607931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:33.601661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:33.635943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:33.635552 4952 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:33:33.635943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:33.635631 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 22:33:33.820165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:33.820130 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:33:34.156929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:34.150421 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:34.634366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:34.634327 4952 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:33:34.634696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:34.634659 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 22:33:35.584991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:35.584739 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:35.630842 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:35.630795 4952 patch_prober.go:29] interesting pod/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:33:35.631161 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:35.631122 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 29 22:33:36.649624 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:36.649593 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:33:36.650161 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:36.650006 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:33:36.650803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:36.650780 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:33:37.263013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:37.262967 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:39.008059 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:39.007959 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:40.520010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:40.519950 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:40.720517 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:33:40.720481 4952 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@9-42915-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f904684_fe8c_4e96_befb_229dbf30522f.slice/crio-b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:33:40.892541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:40.892506 4952 generic.go:356] "Generic (PLEG): container finished" podID="9f904684-fe8c-4e96-befb-229dbf30522f" containerID="b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7" exitCode=0 Oct 29 22:33:40.892759 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:40.892584 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" event={"ID":"9f904684-fe8c-4e96-befb-229dbf30522f","Type":"ContainerDied","Data":"b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7"} Oct 29 22:33:40.893181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:40.893161 4952 scope.go:122] "RemoveContainer" containerID="b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7" Oct 29 22:33:40.895636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:40.895606 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:33:40.895746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:40.895685 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:33:40.895746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:40.895719 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:33:41.182376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:41.182281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:41.344047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:41.344010 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:41.903873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:41.903836 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" event={"ID":"9f904684-fe8c-4e96-befb-229dbf30522f","Type":"ContainerStarted","Data":"ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a"} Oct 29 22:33:42.193360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:42.193275 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:33:42.193525 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:42.193461 4952 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:33:42.193525 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:42.193511 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 22:33:42.269659 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:42.269618 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:43.193002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:43.192967 4952 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:33:43.193437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:43.193062 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 22:33:43.999249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:43.999204 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:44.193598 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:44.193564 4952 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:33:44.194050 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:44.193629 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 22:33:45.193071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:45.193033 4952 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:33:45.193274 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:45.193093 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 22:33:45.565275 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:33:45.565182 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:33:45.697927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:45.694845 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:46.192650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:46.192613 4952 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:33:46.192865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:46.192670 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 22:33:47.193141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:47.193104 4952 patch_prober.go:29] interesting pod/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 22:33:47.193550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:47.193163 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 22:33:47.555203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:47.555096 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:48.199400 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:48.199367 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:33:48.199893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:48.199859 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:33:48.200670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:48.200651 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:33:49.233924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:49.233864 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:50.680318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.680276 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg"] Oct 29 22:33:50.686740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.686687 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg"] Oct 29 22:33:50.686988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.686766 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.705534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.705487 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-90f46c\"/\"kube-root-ca.crt\"" Oct 29 22:33:50.705919 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.705872 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-90f46c\"/\"openshift-service-ca.crt\"" Oct 29 22:33:50.717858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.717827 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-90f46c\"/\"default-dockercfg-l5dxd\"" Oct 29 22:33:50.718155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.718134 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-90f46c\"/\"test-model-90f46c-kserve-self-signed-certs\"" Oct 29 22:33:50.780926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.780233 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35243f45-e93c-48e5-aa76-f9cefdb58328-tls-certs\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.780926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.780475 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-model-cache\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.780926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.780562 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-dshm\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.782409 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.782247 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-tmp-dir\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.782570 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.782551 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjsq\" (UniqueName: \"kubernetes.io/projected/35243f45-e93c-48e5-aa76-f9cefdb58328-kube-api-access-dxjsq\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.782642 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.782594 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-home\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.804103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.802696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:50.891930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.889179 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-model-cache\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.891930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.889268 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-dshm\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.891930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.889452 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-tmp-dir\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.891930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.889523 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjsq\" (UniqueName: \"kubernetes.io/projected/35243f45-e93c-48e5-aa76-f9cefdb58328-kube-api-access-dxjsq\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.891930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.889560 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-home\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.891930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.889711 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35243f45-e93c-48e5-aa76-f9cefdb58328-tls-certs\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.908933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.902579 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-tmp-dir\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.908933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.902870 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-model-cache\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.908933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.907676 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-home\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.920349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.917713 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-dshm\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.955971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.947262 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35243f45-e93c-48e5-aa76-f9cefdb58328-tls-certs\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:50.988846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:50.988809 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjsq\" (UniqueName: \"kubernetes.io/projected/35243f45-e93c-48e5-aa76-f9cefdb58328-kube-api-access-dxjsq\") pod \"test-model-90f46c-kserve-855b4d4cb7-p5zjg\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:51.058927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:51.053423 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:51.096961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:51.094066 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:51.752509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:51.752469 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg"] Oct 29 22:33:51.755198 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:33:51.755163 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35243f45_e93c_48e5_aa76_f9cefdb58328.slice/crio-e518edc4ad1751a4082ca97ca12bc567a9893d79a4dff7688ae93655ec224160 WatchSource:0}: Error finding container e518edc4ad1751a4082ca97ca12bc567a9893d79a4dff7688ae93655ec224160: Status 404 returned error can't find the container with id e518edc4ad1751a4082ca97ca12bc567a9893d79a4dff7688ae93655ec224160 Oct 29 22:33:52.440009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:52.439958 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" event={"ID":"35243f45-e93c-48e5-aa76-f9cefdb58328","Type":"ContainerStarted","Data":"7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1"} Oct 29 22:33:52.440229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:52.440026 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" event={"ID":"35243f45-e93c-48e5-aa76-f9cefdb58328","Type":"ContainerStarted","Data":"e518edc4ad1751a4082ca97ca12bc567a9893d79a4dff7688ae93655ec224160"} Oct 29 22:33:52.489362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:52.489307 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" podStartSLOduration=2.489292752 podStartE2EDuration="2.489292752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:52.484764323 +0000 UTC m=+3097.959352741" watchObservedRunningTime="2026-10-29 22:33:52.489292752 +0000 UTC m=+3097.963881187" Oct 29 22:33:53.023041 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:53.014343 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:53.057781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:53.057740 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:33:54.632497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:54.632402 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:54.803165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:54.803126 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:56.429984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:56.419546 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:58.119930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:58.116586 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:58.313124 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:58.313089 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:58.481178 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:58.481084 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:59.786060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:59.786023 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:59.958276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:33:59.958238 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:01.056395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:01.056361 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:34:01.074448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:01.074415 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:34:01.303033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:01.302994 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:34:01.647919 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:01.644659 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:34:01.989982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:01.985584 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:04.312879 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:04.312839 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:04.809045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:04.808988 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" podUID="9cf35311-cf34-4be6-847c-4088a59bf188" containerName="main" probeResult="failure" output="Get \"https://10.132.0.210:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:04.809811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:04.809773 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:04.832797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:04.829806 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:04.929448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:04.929396 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:05.072103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:05.057066 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:05.095062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:05.095016 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 29 22:34:05.134932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:05.115556 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:34:05.155741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:05.155696 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:05.205038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:05.204986 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 29 22:34:05.340135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:05.339000 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:05.363989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:05.363948 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:06.396380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:06.396344 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:06.576882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:06.576844 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:08.327929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:08.322211 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:09.403694 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:09.403652 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:10.336484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:10.336441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:11.919923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:11.919868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:12.086932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:12.086880 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:12.260335 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:12.260244 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:12.433171 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:12.433125 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:14.033926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:14.027117 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:15.529244 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:34:15.529205 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:34:15.689936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:15.689882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:17.297504 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:17.297461 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:19.016052 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:19.016013 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:20.837573 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:20.837535 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:22.556926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:22.555174 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:22.730144 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:22.730101 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:23.123413 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:23.123370 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:23.621571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:23.621536 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:23.918282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:23.918191 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:24.627084 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:24.627037 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:26.327993 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:26.327949 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:28.046338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:28.046299 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:29.732266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:29.732183 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:31.436218 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:31.436175 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:33.040939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:33.038194 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:34.703279 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:34.703240 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:36.434605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:36.434565 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:38.026285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:38.026240 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:39.645333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:39.645297 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:41.238085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:41.235574 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:42.853928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:42.853579 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:44.871929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:44.865574 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:46.100184 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:34:46.100135 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:34:46.361986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.361001 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:46.393053 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.393004 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:46.483944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.479138 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:46.629983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.626516 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:46.658927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.654702 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" podUID="35243f45-e93c-48e5-aa76-f9cefdb58328" containerName="main" probeResult="failure" output="Get \"https://10.132.0.211:8000/ready\": context deadline exceeded" Oct 29 22:34:46.658927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.657278 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 29 22:34:46.692116 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.692069 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:46.699935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.696987 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:46.807275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.807223 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" podUID="9cf35311-cf34-4be6-847c-4088a59bf188" containerName="main" probeResult="failure" output="Get \"https://10.132.0.210:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:46.836038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.835995 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:46.889174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:46.889004 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:47.102410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:47.102347 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:47.116483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:47.116443 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:47.116955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:47.116926 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 29 22:34:47.161099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:47.161002 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:47.204445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:47.204397 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:47.336927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:47.334328 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:47.640935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:47.634722 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:49.538930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:49.535500 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:51.254695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:51.254649 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:53.053927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:53.048532 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:54.828947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:54.827934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:56.496937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:56.479868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:58.222987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:58.222945 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:59.855975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:34:59.855867 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:01.438932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:01.426437 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:03.043445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:03.043405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:04.692932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:04.692483 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:06.352468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:06.352430 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:07.949729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:07.949691 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:09.557111 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:09.557072 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:11.348303 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:11.348262 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:12.486080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:12.486039 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:13.026795 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:13.026757 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:13.482549 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:13.482504 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:13.667918 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:13.667867 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:14.601204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:14.601154 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7"] Oct 29 22:35:14.672289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:14.672257 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:14.850868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:14.850829 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:15.031981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:15.031471 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:15.324765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:15.324642 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:15.535994 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:35:15.535949 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:35:16.951026 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:16.946506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:18.683468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:18.683397 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:19.602586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:19.602527 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" podUID="81383129-aa94-4349-a1c2-3a3a9240be1b" containerName="payload-processing" containerID="cri-o://5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f" gracePeriod=25 Oct 29 22:35:19.934249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:19.934214 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:35:20.003004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.002966 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z"] Oct 29 22:35:20.004390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.004360 4952 cpu_manager.go:414] "containerMap: removing container" podUID="81383129-aa94-4349-a1c2-3a3a9240be1b" containerName="payload-processing" Oct 29 22:35:20.004390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.004388 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81383129-aa94-4349-a1c2-3a3a9240be1b" containerName="payload-processing" Oct 29 22:35:20.004656 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.004640 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="81383129-aa94-4349-a1c2-3a3a9240be1b" containerName="payload-processing" Oct 29 22:35:20.009550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.009526 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.027384 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.027346 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0cb13f-serving-cert\"" Oct 29 22:35:20.034777 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.034750 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z"] Oct 29 22:35:20.071410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.071372 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-kube-api-access-5xg42\" (UniqueName: \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-kube-api-access-5xg42\") pod \"81383129-aa94-4349-a1c2-3a3a9240be1b\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " Oct 29 22:35:20.071603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.071430 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-routing-overlay-volume\") pod \"81383129-aa94-4349-a1c2-3a3a9240be1b\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " Oct 29 22:35:20.071603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.071572 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-plugins-config-volume\") pod \"81383129-aa94-4349-a1c2-3a3a9240be1b\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " Oct 29 22:35:20.072415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.071870 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-routing-overlay-volume" pod "81383129-aa94-4349-a1c2-3a3a9240be1b" (UID: "81383129-aa94-4349-a1c2-3a3a9240be1b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:20.072415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.071940 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-provider-credentials\") pod \"81383129-aa94-4349-a1c2-3a3a9240be1b\" (UID: \"81383129-aa94-4349-a1c2-3a3a9240be1b\") " Oct 29 22:35:20.072415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.072149 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-plugins-config-volume" pod "81383129-aa94-4349-a1c2-3a3a9240be1b" (UID: "81383129-aa94-4349-a1c2-3a3a9240be1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:20.072415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.072331 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c5b272-9199-4235-b855-4ee209f3f159-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.072415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.072378 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-metrics-certs\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.072729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.072584 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q22v\" (UniqueName: \"kubernetes.io/projected/70c5b272-9199-4235-b855-4ee209f3f159-kube-api-access-2q22v\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.072778 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.072715 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-maas-api-tls\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.073077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.073053 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:35:20.073137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.073085 4952 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81383129-aa94-4349-a1c2-3a3a9240be1b-routing-overlay-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:35:20.075593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.075567 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-provider-credentials" pod "81383129-aa94-4349-a1c2-3a3a9240be1b" (UID: "81383129-aa94-4349-a1c2-3a3a9240be1b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:20.077090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.077064 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-kube-api-access-5xg42" pod "81383129-aa94-4349-a1c2-3a3a9240be1b" (UID: "81383129-aa94-4349-a1c2-3a3a9240be1b"). InnerVolumeSpecName "kube-api-access-5xg42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:20.174094 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.174019 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q22v\" (UniqueName: \"kubernetes.io/projected/70c5b272-9199-4235-b855-4ee209f3f159-kube-api-access-2q22v\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.174243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.174092 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-maas-api-tls\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.174291 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.174252 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c5b272-9199-4235-b855-4ee209f3f159-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.174339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.174288 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-metrics-certs\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.174377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.174361 4952 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-provider-credentials\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:35:20.174377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.174372 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xg42\" (UniqueName: \"kubernetes.io/projected/81383129-aa94-4349-a1c2-3a3a9240be1b-kube-api-access-5xg42\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:35:20.174932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.174889 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c5b272-9199-4235-b855-4ee209f3f159-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.176677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.176654 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-maas-api-tls\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.176775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.176693 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-metrics-certs\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.199953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.199869 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q22v\" (UniqueName: \"kubernetes.io/projected/70c5b272-9199-4235-b855-4ee209f3f159-kube-api-access-2q22v\") pod \"maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.271380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.271340 4952 generic.go:356] "Generic (PLEG): container finished" podID="81383129-aa94-4349-a1c2-3a3a9240be1b" containerID="5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f" exitCode=0 Oct 29 22:35:20.271579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.271390 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" Oct 29 22:35:20.271579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.271425 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" event={"ID":"81383129-aa94-4349-a1c2-3a3a9240be1b","Type":"ContainerDied","Data":"5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f"} Oct 29 22:35:20.271579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.271489 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7" event={"ID":"81383129-aa94-4349-a1c2-3a3a9240be1b","Type":"ContainerDied","Data":"8fe4bb09fbe13113444d7510cc3ba7140bd1e557fd719c3ba08a5857eebbbcd9"} Oct 29 22:35:20.271579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.271547 4952 scope.go:122] "RemoveContainer" containerID="5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f" Oct 29 22:35:20.289129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.289098 4952 scope.go:122] "RemoveContainer" containerID="5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f" Oct 29 22:35:20.289643 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:35:20.289613 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f\": container with ID starting with 5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f not found: ID does not exist" containerID="5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f" Oct 29 22:35:20.289758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.289652 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f"} err="failed to get container status \"5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f\": rpc error: code = NotFound desc = could not find container \"5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f\": container with ID starting with 5c1c18af12dc49b52a79c7cbeec181156fad1b99d6ce432348f61a29dc6ab04f not found: ID does not exist" Oct 29 22:35:20.335512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.335484 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7"] Oct 29 22:35:20.336886 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.336866 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:20.343212 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.343186 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-749f72/payload-processing-external-model-e2e-extmod-a-749f72-7d562ghw7"] Oct 29 22:35:20.401593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.401555 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:20.650479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.650441 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81383129-aa94-4349-a1c2-3a3a9240be1b" path="/var/lib/kubelet/pods/81383129-aa94-4349-a1c2-3a3a9240be1b/volumes" Oct 29 22:35:20.776190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.776153 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z"] Oct 29 22:35:20.777416 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:35:20.777385 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c5b272_9199_4235_b855_4ee209f3f159.slice/crio-894cc42034b67ba18167ecb878d203d036d065a28c1d2a54ff9299d9e34c61bd WatchSource:0}: Error finding container 894cc42034b67ba18167ecb878d203d036d065a28c1d2a54ff9299d9e34c61bd: Status 404 returned error can't find the container with id 894cc42034b67ba18167ecb878d203d036d065a28c1d2a54ff9299d9e34c61bd Oct 29 22:35:20.778444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:20.778423 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:35:21.284656 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:21.284619 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" event={"ID":"70c5b272-9199-4235-b855-4ee209f3f159","Type":"ContainerStarted","Data":"894cc42034b67ba18167ecb878d203d036d065a28c1d2a54ff9299d9e34c61bd"} Oct 29 22:35:21.966928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:21.966767 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:22.296440 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:22.296350 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" event={"ID":"70c5b272-9199-4235-b855-4ee209f3f159","Type":"ContainerStarted","Data":"2fd9515532dc9522c86bb9ff9ef60ddcebdd485b635f2330d83b26104c8dd2dd"} Oct 29 22:35:22.296957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:22.296443 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:22.348476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:22.348421 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" podStartSLOduration=82.88817846 podStartE2EDuration="1m23.348406369s" totalImagesPullingTime="460.227909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:22.343851153 +0000 UTC m=+3187.818439571" watchObservedRunningTime="2026-10-29 22:35:22.348406369 +0000 UTC m=+3187.822994786" Oct 29 22:35:22.703927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:22.700024 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:22.871439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:22.871400 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:23.610936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:23.592680 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:25.355508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:25.355469 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:26.897317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:26.895950 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:28.313743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:28.313706 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:35:28.557926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:28.552298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:29.573348 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:29.573298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:29.745011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:29.744967 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:29.974035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:29.973998 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:30.151840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:30.151794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:30.682928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:30.682537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:31.121207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:31.121168 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:31.949928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:31.944668 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:33.554826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:33.554780 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:35.126778 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:35.126736 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:36.825940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:36.825883 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:38.457005 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:38.454136 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:40.158622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:40.158576 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:41.753989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:41.746346 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:43.494284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:43.494246 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:45.146929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:45.146844 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:45.531201 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:35:45.531108 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:35:46.927688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:46.927644 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:48.585670 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:48.585613 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:50.316688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:50.316648 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:52.128536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:52.128496 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:53.844930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:53.843926 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:55.396056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:55.395882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:55.827931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:55.827876 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:35:56.368282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:56.368231 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:56.545557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:56.545518 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:57.517222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:57.517073 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:59.412924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:35:59.410114 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:01.118189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:01.117978 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:02.382115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:02.382077 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:02.755565 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:02.755450 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:04.426507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:04.425625 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:06.074115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:06.074071 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:07.717803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:07.717682 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:09.491645 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:09.491605 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:11.094164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:11.094124 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:12.598803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:12.598768 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:12.789575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:12.789541 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:14.447930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:14.436705 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:15.531003 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:36:15.530960 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:36:16.133930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:16.129488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:17.702103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:17.702053 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:19.377875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:19.377837 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:20.757326 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:20.757271 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:20.965354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:20.965306 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:22.561933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:22.560646 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:24.220591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:24.220547 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:25.475354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:25.475317 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z"] Oct 29 22:36:25.475880 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:25.475558 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" podUID="70c5b272-9199-4235-b855-4ee209f3f159" containerName="maas-api" containerID="cri-o://2fd9515532dc9522c86bb9ff9ef60ddcebdd485b635f2330d83b26104c8dd2dd" gracePeriod=30 Oct 29 22:36:25.592615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:25.592584 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:26.109527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.109110 4952 generic.go:356] "Generic (PLEG): container finished" podID="70c5b272-9199-4235-b855-4ee209f3f159" containerID="2fd9515532dc9522c86bb9ff9ef60ddcebdd485b635f2330d83b26104c8dd2dd" exitCode=0 Oct 29 22:36:26.109527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.109178 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" event={"ID":"70c5b272-9199-4235-b855-4ee209f3f159","Type":"ContainerDied","Data":"2fd9515532dc9522c86bb9ff9ef60ddcebdd485b635f2330d83b26104c8dd2dd"} Oct 29 22:36:26.146427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.146396 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:36:26.218146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.218108 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs"] Oct 29 22:36:26.219666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.219624 4952 cpu_manager.go:414] "containerMap: removing container" podUID="70c5b272-9199-4235-b855-4ee209f3f159" containerName="maas-api" Oct 29 22:36:26.219788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.219667 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c5b272-9199-4235-b855-4ee209f3f159" containerName="maas-api" Oct 29 22:36:26.220026 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.220000 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c5b272-9199-4235-b855-4ee209f3f159" containerName="maas-api" Oct 29 22:36:26.225165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.225136 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.243072 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.243051 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-27b6db-openshift-default-dockercfg-zqg7l\"" Oct 29 22:36:26.246177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.246117 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:26.252846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.252813 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs"] Oct 29 22:36:26.277609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.277581 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c5b272-9199-4235-b855-4ee209f3f159-kube-api-access-2q22v\" (UniqueName: \"kubernetes.io/projected/70c5b272-9199-4235-b855-4ee209f3f159-kube-api-access-2q22v\") pod \"70c5b272-9199-4235-b855-4ee209f3f159\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " Oct 29 22:36:26.277787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.277693 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-maas-api-tls\") pod \"70c5b272-9199-4235-b855-4ee209f3f159\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " Oct 29 22:36:26.277854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.277836 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c5b272-9199-4235-b855-4ee209f3f159-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c5b272-9199-4235-b855-4ee209f3f159-openshift-service-ca\") pod \"70c5b272-9199-4235-b855-4ee209f3f159\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " Oct 29 22:36:26.278061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.278039 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-metrics-certs\") pod \"70c5b272-9199-4235-b855-4ee209f3f159\" (UID: \"70c5b272-9199-4235-b855-4ee209f3f159\") " Oct 29 22:36:26.278355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.278323 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-certs\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.278544 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.278520 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-token\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.278618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.278558 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d2c654-5589-4747-986c-94ec4fd16ffd-istiod-ca-cert\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.278683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.278663 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfw5r\" (UniqueName: \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-kube-api-access-rfw5r\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.278769 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.278712 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-envoy\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.278834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.278795 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-credential-socket\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.279310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.278895 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-data\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.279310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.279074 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-podinfo\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.279310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.279109 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-socket\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.279599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.279576 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c5b272-9199-4235-b855-4ee209f3f159-openshift-service-ca" pod "70c5b272-9199-4235-b855-4ee209f3f159" (UID: "70c5b272-9199-4235-b855-4ee209f3f159"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:26.280454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.280428 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-metrics-certs" pod "70c5b272-9199-4235-b855-4ee209f3f159" (UID: "70c5b272-9199-4235-b855-4ee209f3f159"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:36:26.281588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.281560 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c5b272-9199-4235-b855-4ee209f3f159-kube-api-access-2q22v" pod "70c5b272-9199-4235-b855-4ee209f3f159" (UID: "70c5b272-9199-4235-b855-4ee209f3f159"). InnerVolumeSpecName "kube-api-access-2q22v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:26.282034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.282009 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-maas-api-tls" pod "70c5b272-9199-4235-b855-4ee209f3f159" (UID: "70c5b272-9199-4235-b855-4ee209f3f159"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:36:26.380816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.380733 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-token\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.380816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.380776 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d2c654-5589-4747-986c-94ec4fd16ffd-istiod-ca-cert\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.380861 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfw5r\" (UniqueName: \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-kube-api-access-rfw5r\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.380917 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-envoy\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381005 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-credential-socket\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381088 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-data\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381279 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381226 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-podinfo\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381272 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-socket\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381327 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-certs\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381531 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c5b272-9199-4235-b855-4ee209f3f159-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:36:26.381624 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381560 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:36:26.381624 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381575 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q22v\" (UniqueName: \"kubernetes.io/projected/70c5b272-9199-4235-b855-4ee209f3f159-kube-api-access-2q22v\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:36:26.381624 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381579 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-data\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381624 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381589 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70c5b272-9199-4235-b855-4ee209f3f159-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:36:26.381790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381645 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-socket\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381657 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d2c654-5589-4747-986c-94ec4fd16ffd-istiod-ca-cert\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381826 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-credential-socket\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.381873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.381841 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-certs\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.383395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.383374 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-envoy\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.384125 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.384108 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-podinfo\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.435968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.434674 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:26.451930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.445656 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfw5r\" (UniqueName: \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-kube-api-access-rfw5r\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.470768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.470498 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-token\") pod \"e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.546484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.546447 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:26.630040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.629998 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:26.881417 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:36:26.881383 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d2c654_5589_4747_986c_94ec4fd16ffd.slice/crio-56a5485600df780ac84815be68ec64d360b4f0cca29261c014d69dcdf258ab90 WatchSource:0}: Error finding container 56a5485600df780ac84815be68ec64d360b4f0cca29261c014d69dcdf258ab90: Status 404 returned error can't find the container with id 56a5485600df780ac84815be68ec64d360b4f0cca29261c014d69dcdf258ab90 Oct 29 22:36:26.881659 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.881637 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs"] Oct 29 22:36:26.885618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.885574 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:36:26.885757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.885666 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:36:26.885757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.885710 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:36:26.917797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:26.917760 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:27.121050 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.121022 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" Oct 29 22:36:27.121245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.121021 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z" event={"ID":"70c5b272-9199-4235-b855-4ee209f3f159","Type":"ContainerDied","Data":"894cc42034b67ba18167ecb878d203d036d065a28c1d2a54ff9299d9e34c61bd"} Oct 29 22:36:27.121245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.121139 4952 scope.go:122] "RemoveContainer" containerID="2fd9515532dc9522c86bb9ff9ef60ddcebdd485b635f2330d83b26104c8dd2dd" Oct 29 22:36:27.124587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.124441 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" event={"ID":"85d2c654-5589-4747-986c-94ec4fd16ffd","Type":"ContainerStarted","Data":"dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8"} Oct 29 22:36:27.124587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.124522 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" event={"ID":"85d2c654-5589-4747-986c-94ec4fd16ffd","Type":"ContainerStarted","Data":"56a5485600df780ac84815be68ec64d360b4f0cca29261c014d69dcdf258ab90"} Oct 29 22:36:27.182081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.181992 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z"] Oct 29 22:36:27.239266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.239228 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cb13f-69f95559b7-pz99z"] Oct 29 22:36:27.250877 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.250766 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" podStartSLOduration=57.250743425 podStartE2EDuration="57.250743425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:27.247435105 +0000 UTC m=+3252.722023523" watchObservedRunningTime="2026-10-29 22:36:27.250743425 +0000 UTC m=+3252.725331839" Oct 29 22:36:27.547249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.547163 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:27.549194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.549156 4952 patch_prober.go:29] interesting pod/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 22:36:27.549358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:27.549216 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 22:36:28.459930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:28.459676 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:28.553317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:28.553285 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:28.649245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:28.649205 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c5b272-9199-4235-b855-4ee209f3f159" path="/var/lib/kubelet/pods/70c5b272-9199-4235-b855-4ee209f3f159/volumes" Oct 29 22:36:29.145524 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:29.145488 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:29.146548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:29.146522 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:36:29.799649 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:29.799614 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:29.974255 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:29.974220 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:30.205854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:30.205819 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:31.505298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:31.505256 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:31.692650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:31.692610 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:32.310025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:32.307694 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:33.761392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:33.761350 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:33.952232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:33.952185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:34.127390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:34.127333 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:35.550159 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:35.547474 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:37.278664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:37.278610 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:39.042675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:39.042572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:41.838054 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:41.838013 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:42.628136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:42.628091 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:44.515232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:44.515196 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:45.531331 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:36:45.531292 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:36:46.323927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:46.323592 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:46.800933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:46.800825 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:47.018049 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:47.018014 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:48.164213 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:48.164171 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:49.734950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:49.734895 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:51.368983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:51.368940 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:51.529984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:51.529946 4952 generic.go:356] "Generic (PLEG): container finished" podID="68024384-a51d-40b9-bb47-d4e2baac27d1" containerID="ae0fc47802a892d86790e808c8275e8956b800e6f4ca5117fd9ca082d057a1b3" exitCode=137 Oct 29 22:36:51.530164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:51.530019 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" event={"ID":"68024384-a51d-40b9-bb47-d4e2baac27d1","Type":"ContainerDied","Data":"ae0fc47802a892d86790e808c8275e8956b800e6f4ca5117fd9ca082d057a1b3"} Oct 29 22:36:51.530528 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:51.530512 4952 scope.go:122] "RemoveContainer" containerID="ae0fc47802a892d86790e808c8275e8956b800e6f4ca5117fd9ca082d057a1b3" Oct 29 22:36:52.541478 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:52.541442 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" event={"ID":"68024384-a51d-40b9-bb47-d4e2baac27d1","Type":"ContainerStarted","Data":"91aadff999f9983ff48ef268d669586245c91b3fed042119830719f88d45d891"} Oct 29 22:36:52.542008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:52.541707 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 22:36:53.017404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:53.017352 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:54.731111 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:54.731055 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:56.450339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:56.441962 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:58.099500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:58.099454 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:59.728930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:36:59.719612 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:01.357459 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:01.357421 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:02.308390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:02.308353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:02.474353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:02.474312 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:03.037640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:03.037602 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:03.553522 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:03.553484 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hj67h" Oct 29 22:37:04.762571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:04.762530 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:06.346927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:06.346625 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:08.184023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:08.183983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:09.736778 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:09.734044 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:10.087698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:10.087651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:10.266408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:10.266367 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:10.943839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:10.943801 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm"] Oct 29 22:37:10.944355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:10.944033 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" podUID="a8883ff6-916d-43bd-a221-c8cea4992968" containerName="maas-api" containerID="cri-o://c9c14914614a840b89c1a46b2e511f3ece3d2d88bbbea4f36d76e41a41047eed" gracePeriod=30 Oct 29 22:37:11.055035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:11.054980 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:11.846825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:11.846499 4952 generic.go:356] "Generic (PLEG): container finished" podID="a8883ff6-916d-43bd-a221-c8cea4992968" containerID="c9c14914614a840b89c1a46b2e511f3ece3d2d88bbbea4f36d76e41a41047eed" exitCode=0 Oct 29 22:37:11.846825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:11.846744 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" event={"ID":"a8883ff6-916d-43bd-a221-c8cea4992968","Type":"ContainerDied","Data":"c9c14914614a840b89c1a46b2e511f3ece3d2d88bbbea4f36d76e41a41047eed"} Oct 29 22:37:11.867768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:11.867401 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:37:11.989593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:11.989554 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4"] Oct 29 22:37:11.995112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:11.992084 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a8883ff6-916d-43bd-a221-c8cea4992968" containerName="maas-api" Oct 29 22:37:11.995112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:11.992109 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8883ff6-916d-43bd-a221-c8cea4992968" containerName="maas-api" Oct 29 22:37:11.995112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:11.992428 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8883ff6-916d-43bd-a221-c8cea4992968" containerName="maas-api" Oct 29 22:37:11.998168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:11.998143 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.001632 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.001572 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4"] Oct 29 22:37:12.015672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.015638 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-52429a-serving-cert\"" Oct 29 22:37:12.042839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.039790 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8883ff6-916d-43bd-a221-c8cea4992968-kube-api-access-rfspl\" (UniqueName: \"kubernetes.io/projected/a8883ff6-916d-43bd-a221-c8cea4992968-kube-api-access-rfspl\") pod \"a8883ff6-916d-43bd-a221-c8cea4992968\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " Oct 29 22:37:12.042839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.039843 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8883ff6-916d-43bd-a221-c8cea4992968-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8883ff6-916d-43bd-a221-c8cea4992968-openshift-service-ca\") pod \"a8883ff6-916d-43bd-a221-c8cea4992968\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " Oct 29 22:37:12.042839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.040069 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-metrics-certs\") pod \"a8883ff6-916d-43bd-a221-c8cea4992968\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " Oct 29 22:37:12.042839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.040147 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-maas-api-tls\") pod \"a8883ff6-916d-43bd-a221-c8cea4992968\" (UID: \"a8883ff6-916d-43bd-a221-c8cea4992968\") " Oct 29 22:37:12.056114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.055167 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8883ff6-916d-43bd-a221-c8cea4992968-openshift-service-ca" pod "a8883ff6-916d-43bd-a221-c8cea4992968" (UID: "a8883ff6-916d-43bd-a221-c8cea4992968"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:12.057065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.057039 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-maas-api-tls" pod "a8883ff6-916d-43bd-a221-c8cea4992968" (UID: "a8883ff6-916d-43bd-a221-c8cea4992968"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:12.057235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.057210 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-metrics-certs" pod "a8883ff6-916d-43bd-a221-c8cea4992968" (UID: "a8883ff6-916d-43bd-a221-c8cea4992968"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:12.059473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.059451 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8883ff6-916d-43bd-a221-c8cea4992968-kube-api-access-rfspl" pod "a8883ff6-916d-43bd-a221-c8cea4992968" (UID: "a8883ff6-916d-43bd-a221-c8cea4992968"). InnerVolumeSpecName "kube-api-access-rfspl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:12.103942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.103879 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:12.142039 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.142006 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-metrics-certs\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.142227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.142100 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ss2k\" (UniqueName: \"kubernetes.io/projected/52678853-aee2-46d9-ac6c-1462743c2e5c-kube-api-access-2ss2k\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.142296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.142231 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-maas-api-tls\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.142351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.142328 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52678853-aee2-46d9-ac6c-1462743c2e5c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.142514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.142494 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfspl\" (UniqueName: \"kubernetes.io/projected/a8883ff6-916d-43bd-a221-c8cea4992968-kube-api-access-rfspl\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:37:12.142514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.142512 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8883ff6-916d-43bd-a221-c8cea4992968-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:37:12.142677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.142528 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:37:12.142677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.142541 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8883ff6-916d-43bd-a221-c8cea4992968-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:37:12.244473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.244244 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-maas-api-tls\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.244473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.244358 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52678853-aee2-46d9-ac6c-1462743c2e5c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.244744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.244534 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-metrics-certs\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.244744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.244576 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ss2k\" (UniqueName: \"kubernetes.io/projected/52678853-aee2-46d9-ac6c-1462743c2e5c-kube-api-access-2ss2k\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.245957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.245928 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52678853-aee2-46d9-ac6c-1462743c2e5c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.248851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.248823 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-maas-api-tls\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.249679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.249656 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-metrics-certs\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.271231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.271162 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ss2k\" (UniqueName: \"kubernetes.io/projected/52678853-aee2-46d9-ac6c-1462743c2e5c-kube-api-access-2ss2k\") pod \"maas-api-e2e-inf-a-52429a-7bd98c45-sltn4\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.278679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.278644 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:12.316974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.316942 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:12.448859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.448818 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:12.717561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.717521 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:12.718069 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:37:12.718039 4952 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8883ff6_916d_43bd_a221_c8cea4992968.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8883ff6_916d_43bd_a221_c8cea4992968.slice/crio-2635f655f4b2198a167c87ca80c7127f3df59e60023ebfa8454041f9e4ae4bd5\": RecentStats: unable to find data in memory cache]" Oct 29 22:37:12.720203 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:37:12.720180 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52678853_aee2_46d9_ac6c_1462743c2e5c.slice/crio-22b67d3b2d4ff03dab3636fc562295e97435f4350907ab049b558ff8374a2c12 WatchSource:0}: Error finding container 22b67d3b2d4ff03dab3636fc562295e97435f4350907ab049b558ff8374a2c12: Status 404 returned error can't find the container with id 22b67d3b2d4ff03dab3636fc562295e97435f4350907ab049b558ff8374a2c12 Oct 29 22:37:12.721483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.721439 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4"] Oct 29 22:37:12.857598 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.857557 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" event={"ID":"52678853-aee2-46d9-ac6c-1462743c2e5c","Type":"ContainerStarted","Data":"22b67d3b2d4ff03dab3636fc562295e97435f4350907ab049b558ff8374a2c12"} Oct 29 22:37:12.859138 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.859112 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" Oct 29 22:37:12.859284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.859129 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm" event={"ID":"a8883ff6-916d-43bd-a221-c8cea4992968","Type":"ContainerDied","Data":"2635f655f4b2198a167c87ca80c7127f3df59e60023ebfa8454041f9e4ae4bd5"} Oct 29 22:37:12.859284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.859190 4952 scope.go:122] "RemoveContainer" containerID="c9c14914614a840b89c1a46b2e511f3ece3d2d88bbbea4f36d76e41a41047eed" Oct 29 22:37:12.885071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.885040 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:12.924817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.924721 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm"] Oct 29 22:37:12.934852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:12.934824 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9ed94-85894f645f-6ncvm"] Oct 29 22:37:13.871215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:13.871178 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" event={"ID":"52678853-aee2-46d9-ac6c-1462743c2e5c","Type":"ContainerStarted","Data":"f9cbd6adb533eadc591f7fe87730a707a23d07ab68acfec5f65bb5a0f5299097"} Oct 29 22:37:13.871738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:13.871240 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:13.925804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:13.925748 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" podStartSLOduration=182.500997961 podStartE2EDuration="3m2.925734117s" totalImagesPullingTime="424.736156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:13.923643407 +0000 UTC m=+3299.398231825" watchObservedRunningTime="2026-10-29 22:37:13.925734117 +0000 UTC m=+3299.400322552" Oct 29 22:37:14.250073 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:14.249814 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:14.661692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:14.661643 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8883ff6-916d-43bd-a221-c8cea4992968" path="/var/lib/kubelet/pods/a8883ff6-916d-43bd-a221-c8cea4992968/volumes" Oct 29 22:37:15.531132 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:37:15.531090 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:37:15.674209 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:15.674169 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:15.927057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:15.927020 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:16.135323 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:16.135243 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:16.216242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:16.216160 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:37:16.238219 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:16.238191 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:37:17.413401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:17.413365 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:17.618734 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:17.618695 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:18.239941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:18.239844 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:19.889324 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:19.889292 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:20.001073 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:20.001031 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:20.227632 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:20.227546 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:20.471964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:20.465120 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:22.412930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:22.410685 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:24.215500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:24.215461 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:25.898165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:25.898125 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:26.380392 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:26.380354 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:28.192969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:28.191007 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:30.035700 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:30.035653 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:31.525006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:31.524952 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z"] Oct 29 22:37:31.729352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:31.729310 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:31.903685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:31.903642 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:33.798531 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:33.798486 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:35.333930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:35.317691 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:36.525847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:36.525769 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" podUID="ffe5ea27-066f-4f88-9f21-e5e09d289018" containerName="payload-processing" containerID="cri-o://df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358" gracePeriod=25 Oct 29 22:37:36.879790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:36.879752 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:37.047001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:37.046956 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:37.221858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:37.221776 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" podUID="ffe5ea27-066f-4f88-9f21-e5e09d289018" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:37:37.908407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:37.908368 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:38.091929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:38.088468 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:39.193512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:39.193413 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:40.845994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:40.845957 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:42.754789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:42.754750 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:42.944990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:42.942988 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:44.581531 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:44.581496 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:45.199421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:45.199377 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:45.392836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:45.392797 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:45.530142 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:37:45.530046 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:37:45.683925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:45.683866 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:45.903402 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:45.903320 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:47.221739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:47.221693 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" podUID="ffe5ea27-066f-4f88-9f21-e5e09d289018" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:37:47.677199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:47.677100 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:49.066956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:49.066898 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4"] Oct 29 22:37:49.069253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:49.067857 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" podUID="52678853-aee2-46d9-ac6c-1462743c2e5c" containerName="maas-api" containerID="cri-o://f9cbd6adb533eadc591f7fe87730a707a23d07ab68acfec5f65bb5a0f5299097" gracePeriod=30 Oct 29 22:37:49.131724 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:49.131625 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:49.527927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:49.527106 4952 generic.go:356] "Generic (PLEG): container finished" podID="52678853-aee2-46d9-ac6c-1462743c2e5c" containerID="f9cbd6adb533eadc591f7fe87730a707a23d07ab68acfec5f65bb5a0f5299097" exitCode=0 Oct 29 22:37:49.527927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:49.527411 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" event={"ID":"52678853-aee2-46d9-ac6c-1462743c2e5c","Type":"ContainerDied","Data":"f9cbd6adb533eadc591f7fe87730a707a23d07ab68acfec5f65bb5a0f5299097"} Oct 29 22:37:49.883493 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:49.883452 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" podUID="52678853-aee2-46d9-ac6c-1462743c2e5c" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.214:8443/health\": dial tcp 10.132.0.214:8443: connect: connection refused" Oct 29 22:37:49.898619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:49.898578 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:50.080748 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:50.080645 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:50.662585 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:50.662535 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:51.492781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.482739 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:51.549069 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.548851 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:51.560426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.560378 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" Oct 29 22:37:51.562128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.560394 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4" event={"ID":"52678853-aee2-46d9-ac6c-1462743c2e5c","Type":"ContainerDied","Data":"22b67d3b2d4ff03dab3636fc562295e97435f4350907ab049b558ff8374a2c12"} Oct 29 22:37:51.562128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.560733 4952 scope.go:122] "RemoveContainer" containerID="f9cbd6adb533eadc591f7fe87730a707a23d07ab68acfec5f65bb5a0f5299097" Oct 29 22:37:51.603753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.599458 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-maas-api-tls\") pod \"52678853-aee2-46d9-ac6c-1462743c2e5c\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " Oct 29 22:37:51.603753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.599594 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52678853-aee2-46d9-ac6c-1462743c2e5c-kube-api-access-2ss2k\" (UniqueName: \"kubernetes.io/projected/52678853-aee2-46d9-ac6c-1462743c2e5c-kube-api-access-2ss2k\") pod \"52678853-aee2-46d9-ac6c-1462743c2e5c\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " Oct 29 22:37:51.603753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.600010 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52678853-aee2-46d9-ac6c-1462743c2e5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52678853-aee2-46d9-ac6c-1462743c2e5c-openshift-service-ca\") pod \"52678853-aee2-46d9-ac6c-1462743c2e5c\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " Oct 29 22:37:51.603753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.600076 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-metrics-certs\") pod \"52678853-aee2-46d9-ac6c-1462743c2e5c\" (UID: \"52678853-aee2-46d9-ac6c-1462743c2e5c\") " Oct 29 22:37:51.618044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.617980 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52678853-aee2-46d9-ac6c-1462743c2e5c-openshift-service-ca" pod "52678853-aee2-46d9-ac6c-1462743c2e5c" (UID: "52678853-aee2-46d9-ac6c-1462743c2e5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:51.703971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.703934 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52678853-aee2-46d9-ac6c-1462743c2e5c-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:37:51.759991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.759950 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-metrics-certs" pod "52678853-aee2-46d9-ac6c-1462743c2e5c" (UID: "52678853-aee2-46d9-ac6c-1462743c2e5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:51.760189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.760088 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-maas-api-tls" pod "52678853-aee2-46d9-ac6c-1462743c2e5c" (UID: "52678853-aee2-46d9-ac6c-1462743c2e5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:51.760189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.760166 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52678853-aee2-46d9-ac6c-1462743c2e5c-kube-api-access-2ss2k" pod "52678853-aee2-46d9-ac6c-1462743c2e5c" (UID: "52678853-aee2-46d9-ac6c-1462743c2e5c"). InnerVolumeSpecName "kube-api-access-2ss2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:51.805859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.805775 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:37:51.805859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.805812 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52678853-aee2-46d9-ac6c-1462743c2e5c-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:37:51.805859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.805825 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ss2k\" (UniqueName: \"kubernetes.io/projected/52678853-aee2-46d9-ac6c-1462743c2e5c-kube-api-access-2ss2k\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:37:52.004750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.915360 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4"] Oct 29 22:37:52.004750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:51.929157 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52429a-7bd98c45-sltn4"] Oct 29 22:37:52.647379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:52.647341 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52678853-aee2-46d9-ac6c-1462743c2e5c" path="/var/lib/kubelet/pods/52678853-aee2-46d9-ac6c-1462743c2e5c/volumes" Oct 29 22:37:52.775175 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:52.775142 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99"] Oct 29 22:37:52.778955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:52.776201 4952 cpu_manager.go:414] "containerMap: removing container" podUID="52678853-aee2-46d9-ac6c-1462743c2e5c" containerName="maas-api" Oct 29 22:37:52.778955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:52.776231 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52678853-aee2-46d9-ac6c-1462743c2e5c" containerName="maas-api" Oct 29 22:37:52.778955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:52.776440 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="52678853-aee2-46d9-ac6c-1462743c2e5c" containerName="maas-api" Oct 29 22:37:52.990830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:52.990733 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99"] Oct 29 22:37:53.031287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:52.990865 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.047559 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.047784 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.047835 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9980446-05cb-46a2-961f-963f0313d496-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.047886 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.047999 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9980446-05cb-46a2-961f-963f0313d496-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.048112 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8ms\" (UniqueName: \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-kube-api-access-wd8ms\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.048166 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.048218 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.048249 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.058116 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.149760 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.149835 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.150049 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.150166 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.150201 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9980446-05cb-46a2-961f-963f0313d496-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.150246 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.150288 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9980446-05cb-46a2-961f-963f0313d496-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.150370 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8ms\" (UniqueName: \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-kube-api-access-wd8ms\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.150421 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.150715 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.150813 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-certs\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.151540 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-credential-socket\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.152059 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-data\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.168698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.152239 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9980446-05cb-46a2-961f-963f0313d496-istiod-ca-cert\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.242085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.204072 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-istio-token\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.242085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.204443 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8ms\" (UniqueName: \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-kube-api-access-wd8ms\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.242085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.205247 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-envoy\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.242085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.205549 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9980446-05cb-46a2-961f-963f0313d496-istio-podinfo\") pod \"e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.299845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.247869 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:53.450384 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.445527 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:53.450384 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.446589 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:53.656928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:53.653399 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:55.000232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:54.973436 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99"] Oct 29 22:37:55.019174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:55.019132 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:37:55.019322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:55.019259 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:37:55.019322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:55.019306 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:37:55.252996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:55.248926 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:55.689405 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:55.689366 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" event={"ID":"b9980446-05cb-46a2-961f-963f0313d496","Type":"ContainerStarted","Data":"af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f"} Oct 29 22:37:55.689660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:55.689430 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" event={"ID":"b9980446-05cb-46a2-961f-963f0313d496","Type":"ContainerStarted","Data":"091b7b7334581c320f0023b18302bb29ebcde1c895ab58378bd2af9a52abbd14"} Oct 29 22:37:55.742311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:55.742254 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" podStartSLOduration=3.742238377 podStartE2EDuration="3.742238377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:55.74205341 +0000 UTC m=+3341.216641828" watchObservedRunningTime="2026-10-29 22:37:55.742238377 +0000 UTC m=+3341.216826795" Oct 29 22:37:56.342246 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:56.342072 4952 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:37:56.342246 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:56.342145 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" podUID="e7dd3058-c239-41cc-88dc-5d492be03792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:37:56.447716 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:56.447659 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:37:56.450081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:56.450037 4952 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 29 22:37:56.450230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:56.450104 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" podUID="b9980446-05cb-46a2-961f-963f0313d496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 29 22:37:57.040423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.040385 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:37:57.178004 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.177964 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe5ea27-066f-4f88-9f21-e5e09d289018-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe5ea27-066f-4f88-9f21-e5e09d289018-plugins-config-volume\") pod \"ffe5ea27-066f-4f88-9f21-e5e09d289018\" (UID: \"ffe5ea27-066f-4f88-9f21-e5e09d289018\") " Oct 29 22:37:57.178199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.178108 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe5ea27-066f-4f88-9f21-e5e09d289018-kube-api-access-r4zh9\" (UniqueName: \"kubernetes.io/projected/ffe5ea27-066f-4f88-9f21-e5e09d289018-kube-api-access-r4zh9\") pod \"ffe5ea27-066f-4f88-9f21-e5e09d289018\" (UID: \"ffe5ea27-066f-4f88-9f21-e5e09d289018\") " Oct 29 22:37:57.178969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.178938 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe5ea27-066f-4f88-9f21-e5e09d289018-plugins-config-volume" pod "ffe5ea27-066f-4f88-9f21-e5e09d289018" (UID: "ffe5ea27-066f-4f88-9f21-e5e09d289018"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:57.181141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.181111 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe5ea27-066f-4f88-9f21-e5e09d289018-kube-api-access-r4zh9" pod "ffe5ea27-066f-4f88-9f21-e5e09d289018" (UID: "ffe5ea27-066f-4f88-9f21-e5e09d289018"). InnerVolumeSpecName "kube-api-access-r4zh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:57.294979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.294115 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe5ea27-066f-4f88-9f21-e5e09d289018-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:37:57.294979 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.294155 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4zh9\" (UniqueName: \"kubernetes.io/projected/ffe5ea27-066f-4f88-9f21-e5e09d289018-kube-api-access-r4zh9\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:37:57.330132 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.330087 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:57.447310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.447268 4952 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 29 22:37:57.447983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.447335 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" podUID="b9980446-05cb-46a2-961f-963f0313d496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 29 22:37:57.711437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.711401 4952 generic.go:356] "Generic (PLEG): container finished" podID="ffe5ea27-066f-4f88-9f21-e5e09d289018" containerID="df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358" exitCode=0 Oct 29 22:37:57.711646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.711443 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" Oct 29 22:37:57.711646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.711477 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" event={"ID":"ffe5ea27-066f-4f88-9f21-e5e09d289018","Type":"ContainerDied","Data":"df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358"} Oct 29 22:37:57.711646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.711560 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z" event={"ID":"ffe5ea27-066f-4f88-9f21-e5e09d289018","Type":"ContainerDied","Data":"16fea3efcf08cd380130e00fd24359774e0036b3d00356a13ac4b87befb4b724"} Oct 29 22:37:57.711646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.711590 4952 scope.go:122] "RemoveContainer" containerID="df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358" Oct 29 22:37:57.745674 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.745651 4952 scope.go:122] "RemoveContainer" containerID="df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358" Oct 29 22:37:57.746062 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:37:57.746025 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358\": container with ID starting with df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358 not found: ID does not exist" containerID="df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358" Oct 29 22:37:57.768866 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.746063 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358"} err="failed to get container status \"df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358\": rpc error: code = NotFound desc = could not find container \"df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358\": container with ID starting with df2dd655f0e3aef2f1034fc128513973cba7e03831453ed2288f0330bdabb358 not found: ID does not exist" Oct 29 22:37:57.773216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.773184 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z"] Oct 29 22:37:57.837627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.785896 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-adfb9d-597fb85f4-j967z"] Oct 29 22:37:57.945929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:57.944871 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:58.122928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:58.122346 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:58.327471 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:58.327101 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99"] Oct 29 22:37:58.327471 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:58.327189 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh"] Oct 29 22:37:58.345551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:58.335113 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" podUID="e7dd3058-c239-41cc-88dc-5d492be03792" containerName="istio-proxy" containerID="cri-o://8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a" gracePeriod=30 Oct 29 22:37:58.444092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:58.428945 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:58.452081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:58.448187 4952 patch_prober.go:29] interesting pod/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 29 22:37:58.452081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:58.448289 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" podUID="b9980446-05cb-46a2-961f-963f0313d496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 29 22:37:58.881576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:58.881502 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" podUID="b9980446-05cb-46a2-961f-963f0313d496" containerName="istio-proxy" containerID="cri-o://af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f" gracePeriod=30 Oct 29 22:37:58.892071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:58.889266 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe5ea27-066f-4f88-9f21-e5e09d289018" path="/var/lib/kubelet/pods/ffe5ea27-066f-4f88-9f21-e5e09d289018/volumes" Oct 29 22:37:59.563483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:37:59.561660 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:00.468077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.468044 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:38:00.575023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.574130 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7dd3058-c239-41cc-88dc-5d492be03792-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dd3058-c239-41cc-88dc-5d492be03792-istiod-ca-cert\") pod \"e7dd3058-c239-41cc-88dc-5d492be03792\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " Oct 29 22:38:00.575023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.574209 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-certs\") pod \"e7dd3058-c239-41cc-88dc-5d492be03792\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " Oct 29 22:38:00.575023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.574248 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-istio-token\" (UniqueName: \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-istio-token\") pod \"e7dd3058-c239-41cc-88dc-5d492be03792\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " Oct 29 22:38:00.575023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.574355 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-envoy\") pod \"e7dd3058-c239-41cc-88dc-5d492be03792\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " Oct 29 22:38:00.575023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.574496 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-credential-socket\") pod \"e7dd3058-c239-41cc-88dc-5d492be03792\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " Oct 29 22:38:00.575023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.574541 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-socket\") pod \"e7dd3058-c239-41cc-88dc-5d492be03792\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " Oct 29 22:38:00.575023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.574583 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-kube-api-access-s9dbs\" (UniqueName: \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-kube-api-access-s9dbs\") pod \"e7dd3058-c239-41cc-88dc-5d492be03792\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " Oct 29 22:38:00.575023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.574623 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7dd3058-c239-41cc-88dc-5d492be03792-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dd3058-c239-41cc-88dc-5d492be03792-istio-podinfo\") pod \"e7dd3058-c239-41cc-88dc-5d492be03792\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " Oct 29 22:38:00.575023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.574693 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-data\") pod \"e7dd3058-c239-41cc-88dc-5d492be03792\" (UID: \"e7dd3058-c239-41cc-88dc-5d492be03792\") " Oct 29 22:38:00.579051 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.579014 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-data" pod "e7dd3058-c239-41cc-88dc-5d492be03792" (UID: "e7dd3058-c239-41cc-88dc-5d492be03792"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:00.579465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.579440 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7dd3058-c239-41cc-88dc-5d492be03792-istiod-ca-cert" pod "e7dd3058-c239-41cc-88dc-5d492be03792" (UID: "e7dd3058-c239-41cc-88dc-5d492be03792"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:00.579745 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.579716 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-certs" pod "e7dd3058-c239-41cc-88dc-5d492be03792" (UID: "e7dd3058-c239-41cc-88dc-5d492be03792"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:00.579974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.579950 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-credential-socket" pod "e7dd3058-c239-41cc-88dc-5d492be03792" (UID: "e7dd3058-c239-41cc-88dc-5d492be03792"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:00.580142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.579961 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-socket" pod "e7dd3058-c239-41cc-88dc-5d492be03792" (UID: "e7dd3058-c239-41cc-88dc-5d492be03792"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:00.584047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.583061 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-kube-api-access-s9dbs" pod "e7dd3058-c239-41cc-88dc-5d492be03792" (UID: "e7dd3058-c239-41cc-88dc-5d492be03792"). InnerVolumeSpecName "kube-api-access-s9dbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:00.584741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.584718 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-istio-token" pod "e7dd3058-c239-41cc-88dc-5d492be03792" (UID: "e7dd3058-c239-41cc-88dc-5d492be03792"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:00.586117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.586053 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7dd3058-c239-41cc-88dc-5d492be03792-istio-podinfo" pod "e7dd3058-c239-41cc-88dc-5d492be03792" (UID: "e7dd3058-c239-41cc-88dc-5d492be03792"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:00.587372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.587237 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-envoy" pod "e7dd3058-c239-41cc-88dc-5d492be03792" (UID: "e7dd3058-c239-41cc-88dc-5d492be03792"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:00.677062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.677028 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dd3058-c239-41cc-88dc-5d492be03792-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.677062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.677062 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.677286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.677077 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dd3058-c239-41cc-88dc-5d492be03792-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.677286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.677090 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.677286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.677103 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.677286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.677115 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.677286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.677127 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.677286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.677143 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dd3058-c239-41cc-88dc-5d492be03792-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.677286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.677157 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9dbs\" (UniqueName: \"kubernetes.io/projected/e7dd3058-c239-41cc-88dc-5d492be03792-kube-api-access-s9dbs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:00.901958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.901861 4952 generic.go:356] "Generic (PLEG): container finished" podID="e7dd3058-c239-41cc-88dc-5d492be03792" containerID="8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a" exitCode=0 Oct 29 22:38:00.901958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.901935 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" event={"ID":"e7dd3058-c239-41cc-88dc-5d492be03792","Type":"ContainerDied","Data":"8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a"} Oct 29 22:38:00.902147 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.901974 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" event={"ID":"e7dd3058-c239-41cc-88dc-5d492be03792","Type":"ContainerDied","Data":"2e115deea538c44ed63c194d2ae2bf9e8975bc8d863e830fee1ff0e3b0fad2a4"} Oct 29 22:38:00.902147 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.901995 4952 scope.go:122] "RemoveContainer" containerID="8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a" Oct 29 22:38:00.902147 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.901937 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh" Oct 29 22:38:00.916723 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.916701 4952 scope.go:122] "RemoveContainer" containerID="8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a" Oct 29 22:38:00.917079 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:00.917049 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a\": container with ID starting with 8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a not found: ID does not exist" containerID="8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a" Oct 29 22:38:00.917196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.917084 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a"} err="failed to get container status \"8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a\": rpc error: code = NotFound desc = could not find container \"8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a\": container with ID starting with 8e2b913f646cd979817eeb4cf45cd3f50e6efd5a76ca20e4e2e488edb2195b2a not found: ID does not exist" Oct 29 22:38:00.967501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.967467 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh"] Oct 29 22:38:00.981181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:00.981151 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-7pdgh"] Oct 29 22:38:01.484928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:01.483952 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:02.669629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:02.669595 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7dd3058-c239-41cc-88dc-5d492be03792" path="/var/lib/kubelet/pods/e7dd3058-c239-41cc-88dc-5d492be03792/volumes" Oct 29 22:38:03.583013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:03.582890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:05.287981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:05.286720 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:05.897952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:05.897920 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz"] Oct 29 22:38:06.156977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:06.153799 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:06.347825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:06.347768 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:07.480620 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:07.480578 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:08.447720 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:08.447682 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:38:09.151861 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.151649 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:09.268410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.268382 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:38:09.389053 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.389017 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-istio-token\" (UniqueName: \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-istio-token\") pod \"b9980446-05cb-46a2-961f-963f0313d496\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " Oct 29 22:38:09.389265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.389066 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-certs\") pod \"b9980446-05cb-46a2-961f-963f0313d496\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " Oct 29 22:38:09.389265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.389207 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9980446-05cb-46a2-961f-963f0313d496-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9980446-05cb-46a2-961f-963f0313d496-istio-podinfo\") pod \"b9980446-05cb-46a2-961f-963f0313d496\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " Oct 29 22:38:09.390282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.389310 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-data\") pod \"b9980446-05cb-46a2-961f-963f0313d496\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " Oct 29 22:38:09.390282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.389434 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-certs" pod "b9980446-05cb-46a2-961f-963f0313d496" (UID: "b9980446-05cb-46a2-961f-963f0313d496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:09.390282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.389584 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-data" pod "b9980446-05cb-46a2-961f-963f0313d496" (UID: "b9980446-05cb-46a2-961f-963f0313d496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:09.390282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.389683 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9980446-05cb-46a2-961f-963f0313d496-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9980446-05cb-46a2-961f-963f0313d496-istiod-ca-cert\") pod \"b9980446-05cb-46a2-961f-963f0313d496\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " Oct 29 22:38:09.390282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.389804 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-credential-socket\") pod \"b9980446-05cb-46a2-961f-963f0313d496\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " Oct 29 22:38:09.390282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.389989 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-envoy\") pod \"b9980446-05cb-46a2-961f-963f0313d496\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " Oct 29 22:38:09.390282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.390044 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-socket\") pod \"b9980446-05cb-46a2-961f-963f0313d496\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " Oct 29 22:38:09.390282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.390100 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-credential-socket" pod "b9980446-05cb-46a2-961f-963f0313d496" (UID: "b9980446-05cb-46a2-961f-963f0313d496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:09.390282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.390145 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9980446-05cb-46a2-961f-963f0313d496-istiod-ca-cert" pod "b9980446-05cb-46a2-961f-963f0313d496" (UID: "b9980446-05cb-46a2-961f-963f0313d496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:09.390282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.390206 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-kube-api-access-wd8ms\" (UniqueName: \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-kube-api-access-wd8ms\") pod \"b9980446-05cb-46a2-961f-963f0313d496\" (UID: \"b9980446-05cb-46a2-961f-963f0313d496\") " Oct 29 22:38:09.391673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.390335 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-socket" pod "b9980446-05cb-46a2-961f-963f0313d496" (UID: "b9980446-05cb-46a2-961f-963f0313d496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:09.391673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.391190 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9980446-05cb-46a2-961f-963f0313d496-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:09.391673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.391214 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:09.391673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.391307 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:09.391673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.391322 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:09.391673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.391336 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:09.392835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.392651 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9980446-05cb-46a2-961f-963f0313d496-istio-podinfo" pod "b9980446-05cb-46a2-961f-963f0313d496" (UID: "b9980446-05cb-46a2-961f-963f0313d496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:09.393012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.392985 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-kube-api-access-wd8ms" pod "b9980446-05cb-46a2-961f-963f0313d496" (UID: "b9980446-05cb-46a2-961f-963f0313d496"). InnerVolumeSpecName "kube-api-access-wd8ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:09.393256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.393237 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-istio-token" pod "b9980446-05cb-46a2-961f-963f0313d496" (UID: "b9980446-05cb-46a2-961f-963f0313d496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:09.393391 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.393347 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-envoy" pod "b9980446-05cb-46a2-961f-963f0313d496" (UID: "b9980446-05cb-46a2-961f-963f0313d496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:09.492550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.492470 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:09.492550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.492500 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9980446-05cb-46a2-961f-963f0313d496-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:09.492550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.492509 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9980446-05cb-46a2-961f-963f0313d496-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:09.492550 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:09.492521 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd8ms\" (UniqueName: \"kubernetes.io/projected/b9980446-05cb-46a2-961f-963f0313d496-kube-api-access-wd8ms\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:10.004699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.004660 4952 generic.go:356] "Generic (PLEG): container finished" podID="b9980446-05cb-46a2-961f-963f0313d496" containerID="af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f" exitCode=0 Oct 29 22:38:10.004921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.004712 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" Oct 29 22:38:10.004921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.004749 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" event={"ID":"b9980446-05cb-46a2-961f-963f0313d496","Type":"ContainerDied","Data":"af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f"} Oct 29 22:38:10.004921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.004807 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99" event={"ID":"b9980446-05cb-46a2-961f-963f0313d496","Type":"ContainerDied","Data":"091b7b7334581c320f0023b18302bb29ebcde1c895ab58378bd2af9a52abbd14"} Oct 29 22:38:10.004921 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.004829 4952 scope.go:122] "RemoveContainer" containerID="af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f" Oct 29 22:38:10.017673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.017654 4952 scope.go:122] "RemoveContainer" containerID="af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f" Oct 29 22:38:10.017972 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:10.017953 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f\": container with ID starting with af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f not found: ID does not exist" containerID="af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f" Oct 29 22:38:10.018029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.017980 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f"} err="failed to get container status \"af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f\": rpc error: code = NotFound desc = could not find container \"af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f\": container with ID starting with af94843f51b93fe56d9b1c894e36cd5a1b93daa0a6ec35723fd30a0b5bd6582f not found: ID does not exist" Oct 29 22:38:10.082151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.082116 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99"] Oct 29 22:38:10.095193 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.095159 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52429a-openshift-default-7ccfff5569-kld99"] Oct 29 22:38:10.647576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.647539 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9980446-05cb-46a2-961f-963f0313d496" path="/var/lib/kubelet/pods/b9980446-05cb-46a2-961f-963f0313d496/volumes" Oct 29 22:38:10.724929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.719636 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:10.905984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:10.898977 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" podUID="21dd7134-a80f-4634-946d-1a23575d964b" containerName="payload-pre-processing" containerID="cri-o://c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0" gracePeriod=25 Oct 29 22:38:11.014099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:11.014064 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:11.202347 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:11.202256 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:12.538925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:12.538794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:12.863061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:12.863003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:13.964371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:13.964330 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:14.500932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:14.455827 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h"] Oct 29 22:38:14.507082 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:14.506533 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" podUID="8b8892a7-74e6-422d-a0df-3d0ece183682" containerName="maas-api" containerID="cri-o://5040a654068cec765be98466a5e3376cd3fe83ec7f36297efdb7b7bd678302bf" gracePeriod=30 Oct 29 22:38:14.699410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:14.694397 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:15.060065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.037196 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:15.093045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.092995 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:38:15.155858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.132006 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:38:15.155858 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.150004 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 29 22:38:15.198731 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.198659 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:38:15.372603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.372512 4952 generic.go:356] "Generic (PLEG): container finished" podID="8b8892a7-74e6-422d-a0df-3d0ece183682" containerID="5040a654068cec765be98466a5e3376cd3fe83ec7f36297efdb7b7bd678302bf" exitCode=0 Oct 29 22:38:15.376836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.376700 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" event={"ID":"8b8892a7-74e6-422d-a0df-3d0ece183682","Type":"ContainerDied","Data":"5040a654068cec765be98466a5e3376cd3fe83ec7f36297efdb7b7bd678302bf"} Oct 29 22:38:15.406730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.386508 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc"] Oct 29 22:38:15.406730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.386812 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" containerID="cri-o://5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f" gracePeriod=30 Oct 29 22:38:15.409020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.408978 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:15.855924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.833072 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" podUID="21dd7134-a80f-4634-946d-1a23575d964b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:38:15.936112 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.936074 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:38:15.945532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:15.944173 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:15.946707 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:15.946661 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:38:16.060201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.060163 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8892a7-74e6-422d-a0df-3d0ece183682-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8892a7-74e6-422d-a0df-3d0ece183682-openshift-service-ca\") pod \"8b8892a7-74e6-422d-a0df-3d0ece183682\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " Oct 29 22:38:16.060682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.060261 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-maas-api-tls\") pod \"8b8892a7-74e6-422d-a0df-3d0ece183682\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " Oct 29 22:38:16.060682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.060292 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8892a7-74e6-422d-a0df-3d0ece183682-kube-api-access-jmvr7\" (UniqueName: \"kubernetes.io/projected/8b8892a7-74e6-422d-a0df-3d0ece183682-kube-api-access-jmvr7\") pod \"8b8892a7-74e6-422d-a0df-3d0ece183682\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " Oct 29 22:38:16.060682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.060592 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8892a7-74e6-422d-a0df-3d0ece183682-openshift-service-ca" pod "8b8892a7-74e6-422d-a0df-3d0ece183682" (UID: "8b8892a7-74e6-422d-a0df-3d0ece183682"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:16.060682 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.060621 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-metrics-certs\") pod \"8b8892a7-74e6-422d-a0df-3d0ece183682\" (UID: \"8b8892a7-74e6-422d-a0df-3d0ece183682\") " Oct 29 22:38:16.061573 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.061451 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8892a7-74e6-422d-a0df-3d0ece183682-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:16.062644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.062614 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-maas-api-tls" pod "8b8892a7-74e6-422d-a0df-3d0ece183682" (UID: "8b8892a7-74e6-422d-a0df-3d0ece183682"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:16.062775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.062719 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-metrics-certs" pod "8b8892a7-74e6-422d-a0df-3d0ece183682" (UID: "8b8892a7-74e6-422d-a0df-3d0ece183682"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:16.063399 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.062996 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8892a7-74e6-422d-a0df-3d0ece183682-kube-api-access-jmvr7" pod "8b8892a7-74e6-422d-a0df-3d0ece183682" (UID: "8b8892a7-74e6-422d-a0df-3d0ece183682"). InnerVolumeSpecName "kube-api-access-jmvr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:16.163566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.163471 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:16.163566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.163517 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b8892a7-74e6-422d-a0df-3d0ece183682-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:16.163566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.163532 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmvr7\" (UniqueName: \"kubernetes.io/projected/8b8892a7-74e6-422d-a0df-3d0ece183682-kube-api-access-jmvr7\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:16.198413 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.198376 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6"] Oct 29 22:38:16.198634 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.198602 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" containerID="cri-o://2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af" gracePeriod=30 Oct 29 22:38:16.350935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.349688 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:16.420227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.420138 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" Oct 29 22:38:16.420494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.420163 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" event={"ID":"8b8892a7-74e6-422d-a0df-3d0ece183682","Type":"ContainerDied","Data":"45873fcdbba0467ff935ffe8f6121d78d5243bc5c0e777f88b646b54c9302803"} Oct 29 22:38:16.420577 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.420534 4952 scope.go:122] "RemoveContainer" containerID="5040a654068cec765be98466a5e3376cd3fe83ec7f36297efdb7b7bd678302bf" Oct 29 22:38:16.482104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.482064 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h"] Oct 29 22:38:16.490124 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.490086 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h"] Oct 29 22:38:16.524622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.524582 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:16.649641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.649599 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8892a7-74e6-422d-a0df-3d0ece183682" path="/var/lib/kubelet/pods/8b8892a7-74e6-422d-a0df-3d0ece183682/volumes" Oct 29 22:38:16.716486 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:16.716354 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:17.124123 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:17.124084 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:18.251931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:18.243395 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:18.573655 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:18.573614 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-50d4af-fbfd7bdcb-sf98h" podUID="8b8892a7-74e6-422d-a0df-3d0ece183682" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.171:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:38:20.222863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:20.222806 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:21.069499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.069468 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:38:21.132362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.132327 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istiod-ca-cert\") pod \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " Oct 29 22:38:21.132534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.132460 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-kube-api-access-5t2dr\" (UniqueName: \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-kube-api-access-5t2dr\") pod \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " Oct 29 22:38:21.132628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.132575 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-envoy\") pod \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " Oct 29 22:38:21.133806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.132678 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-token\" (UniqueName: \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-token\") pod \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " Oct 29 22:38:21.133806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.132737 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istiod-ca-cert" pod "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" (UID: "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:21.133806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.132764 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-certs\") pod \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " Oct 29 22:38:21.133806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.132799 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-data\") pod \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " Oct 29 22:38:21.133806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.132854 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-podinfo\") pod \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " Oct 29 22:38:21.133806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.132887 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-credential-socket\") pod \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " Oct 29 22:38:21.133806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.133034 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-socket\") pod \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\" (UID: \"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a\") " Oct 29 22:38:21.133806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.133360 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-data" pod "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" (UID: "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:21.133806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.133765 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-credential-socket" pod "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" (UID: "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:21.134199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.133844 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-certs" pod "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" (UID: "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:21.134199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.134103 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.134199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.134129 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.134199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.134144 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.134199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.134160 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.134357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.134223 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-socket" pod "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" (UID: "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:21.135626 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.135598 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-envoy" pod "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" (UID: "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:21.135764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.135725 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-token" pod "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" (UID: "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:21.143016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.136709 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-kube-api-access-5t2dr" pod "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" (UID: "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a"). InnerVolumeSpecName "kube-api-access-5t2dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:21.143016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.141081 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-podinfo" pod "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" (UID: "0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:21.236521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.236483 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.236521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.236524 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.236998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.236540 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2dr\" (UniqueName: \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-kube-api-access-5t2dr\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.236998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.236554 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.236998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.236567 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.487366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.487327 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:38:21.540734 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.540696 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-podinfo\") pod \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " Oct 29 22:38:21.540957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.540776 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-socket\") pod \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " Oct 29 22:38:21.540957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.540836 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-token\" (UniqueName: \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-token\") pod \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " Oct 29 22:38:21.541095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.540972 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istiod-ca-cert\") pod \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " Oct 29 22:38:21.541149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.541097 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-credential-socket\") pod \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " Oct 29 22:38:21.541432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.541284 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-envoy\") pod \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " Oct 29 22:38:21.541432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.541324 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-kube-api-access-vgknw\" (UniqueName: \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-kube-api-access-vgknw\") pod \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " Oct 29 22:38:21.541432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.541359 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-certs\") pod \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " Oct 29 22:38:21.543750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.541437 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-data\") pod \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\" (UID: \"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2\") " Oct 29 22:38:21.543750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.542501 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-data" pod "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" (UID: "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:21.543750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.543049 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istiod-ca-cert" pod "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" (UID: "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:21.543750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.543353 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-credential-socket" pod "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" (UID: "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:21.543750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.543486 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-certs" pod "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" (UID: "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:21.544713 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.543983 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-socket" pod "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" (UID: "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:21.547105 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.545617 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-envoy" pod "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" (UID: "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:21.547222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.547136 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-kube-api-access-vgknw" pod "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" (UID: "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2"). InnerVolumeSpecName "kube-api-access-vgknw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:21.549254 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.549229 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-podinfo" pod "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" (UID: "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:21.551181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.551141 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-token" pod "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" (UID: "85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:21.562878 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.562843 4952 generic.go:356] "Generic (PLEG): container finished" podID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerID="2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af" exitCode=0 Oct 29 22:38:21.563045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.562935 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" event={"ID":"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2","Type":"ContainerDied","Data":"2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af"} Oct 29 22:38:21.563045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.562993 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" event={"ID":"85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2","Type":"ContainerDied","Data":"23585ef5ef6b64fd92295bc95193f23c5059955237bc7673e1c78f22ff5b231e"} Oct 29 22:38:21.563045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.562949 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6" Oct 29 22:38:21.563248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.563021 4952 scope.go:122] "RemoveContainer" containerID="2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af" Oct 29 22:38:21.564789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.564709 4952 generic.go:356] "Generic (PLEG): container finished" podID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerID="5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f" exitCode=0 Oct 29 22:38:21.564789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.564771 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" event={"ID":"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a","Type":"ContainerDied","Data":"5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f"} Oct 29 22:38:21.564946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.564806 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" event={"ID":"0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a","Type":"ContainerDied","Data":"be045623ee683658e855b400af432fd1760dd07fb601f4e907c75720fb136f4c"} Oct 29 22:38:21.564946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.564858 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc" Oct 29 22:38:21.581818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.581792 4952 scope.go:122] "RemoveContainer" containerID="2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af" Oct 29 22:38:21.582165 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:21.582144 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af\": container with ID starting with 2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af not found: ID does not exist" containerID="2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af" Oct 29 22:38:21.582266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.582175 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af"} err="failed to get container status \"2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af\": rpc error: code = NotFound desc = could not find container \"2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af\": container with ID starting with 2fa8d362ac59cc49f058218a515e8afb8ea87eef40b793612405b6082d6a97af not found: ID does not exist" Oct 29 22:38:21.582266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.582198 4952 scope.go:122] "RemoveContainer" containerID="5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f" Oct 29 22:38:21.595770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.595748 4952 scope.go:122] "RemoveContainer" containerID="5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f" Oct 29 22:38:21.596223 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:21.596194 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f\": container with ID starting with 5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f not found: ID does not exist" containerID="5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f" Oct 29 22:38:21.596366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.596264 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f"} err="failed to get container status \"5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f\": rpc error: code = NotFound desc = could not find container \"5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f\": container with ID starting with 5f16fd2cbe0f42a5ed91e60711e2d5a40b608ad62038e136f97d4d849da3101f not found: ID does not exist" Oct 29 22:38:21.627603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.627565 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc"] Oct 29 22:38:21.632542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.632511 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8m9fc"] Oct 29 22:38:21.644047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.644013 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.644047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.644047 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgknw\" (UniqueName: \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-kube-api-access-vgknw\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.644298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.644063 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.644298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.644078 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.644298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.644091 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.644298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.644106 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.644298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.644121 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.644298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.644134 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.644298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.644148 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:21.670862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.670813 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6"] Oct 29 22:38:21.678317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:21.678282 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-grwt6"] Oct 29 22:38:22.035949 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:22.034665 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:22.648616 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:22.648576 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" path="/var/lib/kubelet/pods/0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a/volumes" Oct 29 22:38:22.649447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:22.649422 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" path="/var/lib/kubelet/pods/85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2/volumes" Oct 29 22:38:23.625047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:23.625005 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:23.810221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:23.810179 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:24.124929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:24.124190 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:25.221071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:25.221018 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:25.396354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:25.396305 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:25.525983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:25.525093 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm"] Oct 29 22:38:25.525983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:25.525371 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" containerID="cri-o://1b537c510231bde3cce31639dd7a1663adf9dd9dd0c04797fb48ceb093f731db" gracePeriod=30 Oct 29 22:38:25.616042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:25.615998 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" podUID="21dd7134-a80f-4634-946d-1a23575d964b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:38:25.650926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:25.650615 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:25.834657 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:25.834619 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:27.656926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:27.654667 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:28.161788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:28.161741 4952 patch_prober.go:29] interesting pod/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:38:28.161989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:28.161798 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:38:28.944700 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:28.944662 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p"] Oct 29 22:38:28.945204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:28.944935 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" podUID="59b1d6bd-d0af-4db4-9d87-e50a8116a21e" containerName="maas-api" containerID="cri-o://1e6f1592559a66bea57b862ac7dc602962878dc7f4e8757ef25294fb67ffd68e" gracePeriod=30 Oct 29 22:38:29.026667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:29.026627 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:29.650665 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:29.650613 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r"] Oct 29 22:38:29.818792 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:29.818725 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" containerID="cri-o://225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838" gracePeriod=30 Oct 29 22:38:29.851419 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:29.851364 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:29.855247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:29.855214 4952 generic.go:356] "Generic (PLEG): container finished" podID="59b1d6bd-d0af-4db4-9d87-e50a8116a21e" containerID="1e6f1592559a66bea57b862ac7dc602962878dc7f4e8757ef25294fb67ffd68e" exitCode=0 Oct 29 22:38:29.856587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:29.855551 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" event={"ID":"59b1d6bd-d0af-4db4-9d87-e50a8116a21e","Type":"ContainerDied","Data":"1e6f1592559a66bea57b862ac7dc602962878dc7f4e8757ef25294fb67ffd68e"} Oct 29 22:38:29.941918 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:29.941821 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:38:30.010571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.010530 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-maas-api-tls\") pod \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " Oct 29 22:38:30.011084 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.010638 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-kube-api-access-26bk7\" (UniqueName: \"kubernetes.io/projected/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-kube-api-access-26bk7\") pod \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " Oct 29 22:38:30.011084 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.010753 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-openshift-service-ca\") pod \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " Oct 29 22:38:30.011084 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.010930 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-metrics-certs\") pod \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\" (UID: \"59b1d6bd-d0af-4db4-9d87-e50a8116a21e\") " Oct 29 22:38:30.011278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.011160 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-openshift-service-ca" pod "59b1d6bd-d0af-4db4-9d87-e50a8116a21e" (UID: "59b1d6bd-d0af-4db4-9d87-e50a8116a21e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:30.011737 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.011696 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:30.017719 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.017689 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-metrics-certs" pod "59b1d6bd-d0af-4db4-9d87-e50a8116a21e" (UID: "59b1d6bd-d0af-4db4-9d87-e50a8116a21e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:30.017883 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.017858 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-maas-api-tls" pod "59b1d6bd-d0af-4db4-9d87-e50a8116a21e" (UID: "59b1d6bd-d0af-4db4-9d87-e50a8116a21e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:38:30.017989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.017874 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-kube-api-access-26bk7" pod "59b1d6bd-d0af-4db4-9d87-e50a8116a21e" (UID: "59b1d6bd-d0af-4db4-9d87-e50a8116a21e"). InnerVolumeSpecName "kube-api-access-26bk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:30.067215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.067174 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:30.154944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.123750 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26bk7\" (UniqueName: \"kubernetes.io/projected/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-kube-api-access-26bk7\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:30.165190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.162852 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:30.165190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.162888 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b1d6bd-d0af-4db4-9d87-e50a8116a21e-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:30.477380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.477339 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:30.660812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.660705 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:30.850045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.850008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:30.866871 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.866838 4952 generic.go:356] "Generic (PLEG): container finished" podID="3d3ce13d-da21-4593-8eba-603813119eea" containerID="1b537c510231bde3cce31639dd7a1663adf9dd9dd0c04797fb48ceb093f731db" exitCode=0 Oct 29 22:38:30.867034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.866936 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" event={"ID":"3d3ce13d-da21-4593-8eba-603813119eea","Type":"ContainerDied","Data":"1b537c510231bde3cce31639dd7a1663adf9dd9dd0c04797fb48ceb093f731db"} Oct 29 22:38:30.867034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.867014 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" event={"ID":"3d3ce13d-da21-4593-8eba-603813119eea","Type":"ContainerDied","Data":"551a486cb13665e46d77f14b7361ec5d25daba09fbcb2da5ca9608a8bbd5be71"} Oct 29 22:38:30.867211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.867039 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="551a486cb13665e46d77f14b7361ec5d25daba09fbcb2da5ca9608a8bbd5be71" Oct 29 22:38:30.867211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.867057 4952 scope.go:122] "RemoveContainer" containerID="0f4c34fee3479fd25afe0124eedac71bf242f95d818452afd16ccd1a843cbacb" Oct 29 22:38:30.869045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.869023 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" Oct 29 22:38:30.869180 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.869053 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p" event={"ID":"59b1d6bd-d0af-4db4-9d87-e50a8116a21e","Type":"ContainerDied","Data":"38aebf99ebd88b2d4fcfeca623686ee98ef7286829bcfd889054cb7fbe774246"} Oct 29 22:38:30.875460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.875433 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:38:30.880809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.880788 4952 scope.go:122] "RemoveContainer" containerID="1e6f1592559a66bea57b862ac7dc602962878dc7f4e8757ef25294fb67ffd68e" Oct 29 22:38:30.930491 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.930459 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p"] Oct 29 22:38:30.943276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.943246 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0412c3-5dcc765f5-jk95p"] Oct 29 22:38:30.978896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.978855 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-socket\") pod \"3d3ce13d-da21-4593-8eba-603813119eea\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " Oct 29 22:38:30.979113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.978948 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d3ce13d-da21-4593-8eba-603813119eea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3ce13d-da21-4593-8eba-603813119eea-istio-podinfo\") pod \"3d3ce13d-da21-4593-8eba-603813119eea\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " Oct 29 22:38:30.979113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.979102 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3ce13d-da21-4593-8eba-603813119eea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3ce13d-da21-4593-8eba-603813119eea-istiod-ca-cert\") pod \"3d3ce13d-da21-4593-8eba-603813119eea\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " Oct 29 22:38:30.979239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.979137 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-kube-api-access-m2gkr\" (UniqueName: \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-kube-api-access-m2gkr\") pod \"3d3ce13d-da21-4593-8eba-603813119eea\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " Oct 29 22:38:30.979239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.979198 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-data\") pod \"3d3ce13d-da21-4593-8eba-603813119eea\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " Oct 29 22:38:30.979239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.979201 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-socket" pod "3d3ce13d-da21-4593-8eba-603813119eea" (UID: "3d3ce13d-da21-4593-8eba-603813119eea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:30.979483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.979298 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-certs\") pod \"3d3ce13d-da21-4593-8eba-603813119eea\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " Oct 29 22:38:30.979483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.979343 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-istio-token\" (UniqueName: \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-istio-token\") pod \"3d3ce13d-da21-4593-8eba-603813119eea\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " Oct 29 22:38:30.979483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.979367 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-envoy\") pod \"3d3ce13d-da21-4593-8eba-603813119eea\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " Oct 29 22:38:30.979483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.979399 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-credential-socket\") pod \"3d3ce13d-da21-4593-8eba-603813119eea\" (UID: \"3d3ce13d-da21-4593-8eba-603813119eea\") " Oct 29 22:38:30.979763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.979525 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3ce13d-da21-4593-8eba-603813119eea-istiod-ca-cert" pod "3d3ce13d-da21-4593-8eba-603813119eea" (UID: "3d3ce13d-da21-4593-8eba-603813119eea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:30.980057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.980027 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-certs" pod "3d3ce13d-da21-4593-8eba-603813119eea" (UID: "3d3ce13d-da21-4593-8eba-603813119eea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:30.980279 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.980255 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:30.980519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.980285 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:30.980519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.980284 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-data" pod "3d3ce13d-da21-4593-8eba-603813119eea" (UID: "3d3ce13d-da21-4593-8eba-603813119eea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:30.980519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.980300 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3ce13d-da21-4593-8eba-603813119eea-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:30.980519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.980388 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-credential-socket" pod "3d3ce13d-da21-4593-8eba-603813119eea" (UID: "3d3ce13d-da21-4593-8eba-603813119eea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:30.982776 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.982753 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-istio-token" pod "3d3ce13d-da21-4593-8eba-603813119eea" (UID: "3d3ce13d-da21-4593-8eba-603813119eea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:30.983485 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.982838 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-kube-api-access-m2gkr" pod "3d3ce13d-da21-4593-8eba-603813119eea" (UID: "3d3ce13d-da21-4593-8eba-603813119eea"). InnerVolumeSpecName "kube-api-access-m2gkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:30.983485 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:30.982994 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d3ce13d-da21-4593-8eba-603813119eea-istio-podinfo" pod "3d3ce13d-da21-4593-8eba-603813119eea" (UID: "3d3ce13d-da21-4593-8eba-603813119eea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:31.003539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.003439 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-envoy" pod "3d3ce13d-da21-4593-8eba-603813119eea" (UID: "3d3ce13d-da21-4593-8eba-603813119eea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:31.081916 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.081867 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3ce13d-da21-4593-8eba-603813119eea-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.081916 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.081916 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2gkr\" (UniqueName: \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-kube-api-access-m2gkr\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.082382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.081928 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.082382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.081937 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3ce13d-da21-4593-8eba-603813119eea-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.082382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.081945 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.082382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.081953 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ce13d-da21-4593-8eba-603813119eea-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.320133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.317123 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:38:31.403961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.403856 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dd7134-a80f-4634-946d-1a23575d964b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dd7134-a80f-4634-946d-1a23575d964b-plugins-config-volume\") pod \"21dd7134-a80f-4634-946d-1a23575d964b\" (UID: \"21dd7134-a80f-4634-946d-1a23575d964b\") " Oct 29 22:38:31.404169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.404012 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dd7134-a80f-4634-946d-1a23575d964b-kube-api-access-z6nj4\" (UniqueName: \"kubernetes.io/projected/21dd7134-a80f-4634-946d-1a23575d964b-kube-api-access-z6nj4\") pod \"21dd7134-a80f-4634-946d-1a23575d964b\" (UID: \"21dd7134-a80f-4634-946d-1a23575d964b\") " Oct 29 22:38:31.406477 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.406161 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dd7134-a80f-4634-946d-1a23575d964b-plugins-config-volume" pod "21dd7134-a80f-4634-946d-1a23575d964b" (UID: "21dd7134-a80f-4634-946d-1a23575d964b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:31.410542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.408765 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dd7134-a80f-4634-946d-1a23575d964b-kube-api-access-z6nj4" pod "21dd7134-a80f-4634-946d-1a23575d964b" (UID: "21dd7134-a80f-4634-946d-1a23575d964b"). InnerVolumeSpecName "kube-api-access-z6nj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:31.507110 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.507067 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dd7134-a80f-4634-946d-1a23575d964b-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.507110 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.507102 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6nj4\" (UniqueName: \"kubernetes.io/projected/21dd7134-a80f-4634-946d-1a23575d964b-kube-api-access-z6nj4\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:31.888951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.885684 4952 generic.go:356] "Generic (PLEG): container finished" podID="21dd7134-a80f-4634-946d-1a23575d964b" containerID="c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0" exitCode=0 Oct 29 22:38:31.888951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.885889 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" event={"ID":"21dd7134-a80f-4634-946d-1a23575d964b","Type":"ContainerDied","Data":"c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0"} Oct 29 22:38:31.888951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.885958 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" event={"ID":"21dd7134-a80f-4634-946d-1a23575d964b","Type":"ContainerDied","Data":"f557ab8caa7293dbd8891327754edf969d61730210a920649b7e6c4f926872c2"} Oct 29 22:38:31.888951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.885987 4952 scope.go:122] "RemoveContainer" containerID="c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0" Oct 29 22:38:31.888951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.886102 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz" Oct 29 22:38:31.908454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.908303 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm" Oct 29 22:38:31.921659 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.920214 4952 scope.go:122] "RemoveContainer" containerID="c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0" Oct 29 22:38:31.921659 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:31.920622 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0\": container with ID starting with c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0 not found: ID does not exist" containerID="c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0" Oct 29 22:38:31.921659 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:31.920664 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0"} err="failed to get container status \"c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0\": rpc error: code = NotFound desc = could not find container \"c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0\": container with ID starting with c8f516cb463b7fde89bd7a52e8ea6d48355ef8cc179517c1fa757855e3cb26d0 not found: ID does not exist" Oct 29 22:38:32.019613 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:32.019579 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz"] Oct 29 22:38:32.027345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:32.027311 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0412c3-7545f5bcbc-sqxtz"] Oct 29 22:38:32.067589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:32.067549 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm"] Oct 29 22:38:32.074736 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:32.074702 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-50d4af-openshift-default-59db6fb5c-ldwgm"] Oct 29 22:38:32.648018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:32.647972 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21dd7134-a80f-4634-946d-1a23575d964b" path="/var/lib/kubelet/pods/21dd7134-a80f-4634-946d-1a23575d964b/volumes" Oct 29 22:38:32.648665 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:32.648644 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3ce13d-da21-4593-8eba-603813119eea" path="/var/lib/kubelet/pods/3d3ce13d-da21-4593-8eba-603813119eea/volumes" Oct 29 22:38:32.649190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:32.649174 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b1d6bd-d0af-4db4-9d87-e50a8116a21e" path="/var/lib/kubelet/pods/59b1d6bd-d0af-4db4-9d87-e50a8116a21e/volumes" Oct 29 22:38:32.738926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:32.731198 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:34.457411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:34.457373 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:35.008305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.008267 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f"] Oct 29 22:38:35.189454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.189077 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:38:35.222571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.222531 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:35.319625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.319592 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-kube-api-access-xwtg4\" (UniqueName: \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-kube-api-access-xwtg4\") pod \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " Oct 29 22:38:35.319625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.319639 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-credential-socket\") pod \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " Oct 29 22:38:35.320037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.319786 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-envoy\") pod \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " Oct 29 22:38:35.320037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.319945 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-token\" (UniqueName: \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-token\") pod \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " Oct 29 22:38:35.320037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.320016 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-certs\") pod \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " Oct 29 22:38:35.320245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.320072 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-credential-socket" pod "8e3543df-0b4d-4a83-ba52-ed177a1dfb93" (UID: "8e3543df-0b4d-4a83-ba52-ed177a1dfb93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:35.320245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.320195 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istiod-ca-cert\") pod \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " Oct 29 22:38:35.320245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.320220 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-socket\") pod \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " Oct 29 22:38:35.320245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.320247 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-podinfo\") pod \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " Oct 29 22:38:35.320458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.320273 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-data\") pod \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\" (UID: \"8e3543df-0b4d-4a83-ba52-ed177a1dfb93\") " Oct 29 22:38:35.320458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.320370 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-certs" pod "8e3543df-0b4d-4a83-ba52-ed177a1dfb93" (UID: "8e3543df-0b4d-4a83-ba52-ed177a1dfb93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:35.320569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.320544 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-socket" pod "8e3543df-0b4d-4a83-ba52-ed177a1dfb93" (UID: "8e3543df-0b4d-4a83-ba52-ed177a1dfb93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:35.320629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.320570 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istiod-ca-cert" pod "8e3543df-0b4d-4a83-ba52-ed177a1dfb93" (UID: "8e3543df-0b4d-4a83-ba52-ed177a1dfb93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:35.320758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.320740 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-data" pod "8e3543df-0b4d-4a83-ba52-ed177a1dfb93" (UID: "8e3543df-0b4d-4a83-ba52-ed177a1dfb93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:35.321044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.321024 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:35.321044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.321042 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:35.321953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.321051 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:35.321953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.321061 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:35.321953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.321080 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:35.322322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.322302 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-envoy" pod "8e3543df-0b4d-4a83-ba52-ed177a1dfb93" (UID: "8e3543df-0b4d-4a83-ba52-ed177a1dfb93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:35.322758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.322734 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-token" pod "8e3543df-0b4d-4a83-ba52-ed177a1dfb93" (UID: "8e3543df-0b4d-4a83-ba52-ed177a1dfb93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:35.322916 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.322888 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-kube-api-access-xwtg4" pod "8e3543df-0b4d-4a83-ba52-ed177a1dfb93" (UID: "8e3543df-0b4d-4a83-ba52-ed177a1dfb93"). InnerVolumeSpecName "kube-api-access-xwtg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:35.323009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.322987 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-podinfo" pod "8e3543df-0b4d-4a83-ba52-ed177a1dfb93" (UID: "8e3543df-0b4d-4a83-ba52-ed177a1dfb93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:35.391132 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.391096 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:35.445020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.437435 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:35.445020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.437476 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:35.445020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.437490 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwtg4\" (UniqueName: \"kubernetes.io/projected/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-kube-api-access-xwtg4\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:35.445020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.437507 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3543df-0b4d-4a83-ba52-ed177a1dfb93-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:35.964928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.964050 4952 generic.go:356] "Generic (PLEG): container finished" podID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerID="225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838" exitCode=0 Oct 29 22:38:35.964928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.964332 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" event={"ID":"8e3543df-0b4d-4a83-ba52-ed177a1dfb93","Type":"ContainerDied","Data":"225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838"} Oct 29 22:38:35.964928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.964390 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" event={"ID":"8e3543df-0b4d-4a83-ba52-ed177a1dfb93","Type":"ContainerDied","Data":"64a7758aee6e464c48bda02d9d471027c2fe247fc2e8dd14f00a6e8552bb8148"} Oct 29 22:38:35.964928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.964420 4952 scope.go:122] "RemoveContainer" containerID="225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838" Oct 29 22:38:35.964928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.964591 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r" Oct 29 22:38:35.990368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.989340 4952 scope.go:122] "RemoveContainer" containerID="225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838" Oct 29 22:38:35.990368 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:35.989685 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838\": container with ID starting with 225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838 not found: ID does not exist" containerID="225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838" Oct 29 22:38:35.990368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:35.989719 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838"} err="failed to get container status \"225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838\": rpc error: code = NotFound desc = could not find container \"225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838\": container with ID starting with 225e07aebf22c0a21826b63d532e011e9af6e727cc0afc7fd18672eb7bbbb838 not found: ID does not exist" Oct 29 22:38:36.038842 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:36.038803 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r"] Oct 29 22:38:36.050233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:36.050196 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-07e166-openshift-default-7f46fc577c-c5z9r"] Oct 29 22:38:36.352928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:36.345779 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:36.672139 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:36.672045 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" path="/var/lib/kubelet/pods/8e3543df-0b4d-4a83-ba52-ed177a1dfb93/volumes" Oct 29 22:38:36.696933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:36.696887 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8"] Oct 29 22:38:36.874343 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:36.874308 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:37.055759 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:37.055655 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:38.122049 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:38.108983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:38.543982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:38.543858 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:38.718569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:38.718534 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:38.911924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:38.907847 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:39.820799 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:39.820608 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:40.008674 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:40.008628 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" podUID="46cb18bb-e422-46b6-bbea-0881c23bcd7f" containerName="payload-processing" containerID="cri-o://27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04" gracePeriod=25 Oct 29 22:38:40.783035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:40.783005 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:38:40.898427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:40.898338 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cb18bb-e422-46b6-bbea-0881c23bcd7f-kube-api-access-pbv7h\" (UniqueName: \"kubernetes.io/projected/46cb18bb-e422-46b6-bbea-0881c23bcd7f-kube-api-access-pbv7h\") pod \"46cb18bb-e422-46b6-bbea-0881c23bcd7f\" (UID: \"46cb18bb-e422-46b6-bbea-0881c23bcd7f\") " Oct 29 22:38:40.898427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:40.898398 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46cb18bb-e422-46b6-bbea-0881c23bcd7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46cb18bb-e422-46b6-bbea-0881c23bcd7f-plugins-config-volume\") pod \"46cb18bb-e422-46b6-bbea-0881c23bcd7f\" (UID: \"46cb18bb-e422-46b6-bbea-0881c23bcd7f\") " Oct 29 22:38:40.898990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:40.898777 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cb18bb-e422-46b6-bbea-0881c23bcd7f-plugins-config-volume" pod "46cb18bb-e422-46b6-bbea-0881c23bcd7f" (UID: "46cb18bb-e422-46b6-bbea-0881c23bcd7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:40.899346 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:40.899324 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46cb18bb-e422-46b6-bbea-0881c23bcd7f-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.900704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:40.900680 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cb18bb-e422-46b6-bbea-0881c23bcd7f-kube-api-access-pbv7h" pod "46cb18bb-e422-46b6-bbea-0881c23bcd7f" (UID: "46cb18bb-e422-46b6-bbea-0881c23bcd7f"). InnerVolumeSpecName "kube-api-access-pbv7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:40.982126 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:40.982085 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:41.001154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.001114 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbv7h\" (UniqueName: \"kubernetes.io/projected/46cb18bb-e422-46b6-bbea-0881c23bcd7f-kube-api-access-pbv7h\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.163471 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.163250 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:41.171611 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.171580 4952 generic.go:356] "Generic (PLEG): container finished" podID="46cb18bb-e422-46b6-bbea-0881c23bcd7f" containerID="27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04" exitCode=0 Oct 29 22:38:41.171830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.171816 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" event={"ID":"46cb18bb-e422-46b6-bbea-0881c23bcd7f","Type":"ContainerDied","Data":"27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04"} Oct 29 22:38:41.171879 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.171859 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" event={"ID":"46cb18bb-e422-46b6-bbea-0881c23bcd7f","Type":"ContainerDied","Data":"1a44f31e5f9063e9c46a80ac18d2e55f04373a48fdc7d507884199301ae4e040"} Oct 29 22:38:41.171937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.171891 4952 scope.go:122] "RemoveContainer" containerID="27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04" Oct 29 22:38:41.172543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.172039 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f" Oct 29 22:38:41.187407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.187007 4952 scope.go:122] "RemoveContainer" containerID="27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04" Oct 29 22:38:41.187407 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:41.187335 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04\": container with ID starting with 27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04 not found: ID does not exist" containerID="27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04" Oct 29 22:38:41.187407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.187372 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04"} err="failed to get container status \"27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04\": rpc error: code = NotFound desc = could not find container \"27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04\": container with ID starting with 27ddae342896c5227ea91d9058cdbb993b79c453af177f19a6140f0611a6fb04 not found: ID does not exist" Oct 29 22:38:41.234966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.234892 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f"] Oct 29 22:38:41.241486 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.241456 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26abe8-68b5975874-l5q4f"] Oct 29 22:38:41.665320 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.665278 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:41.697806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:41.697762 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" podUID="ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358" containerName="payload-pre-processing" containerID="cri-o://8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e" gracePeriod=25 Oct 29 22:38:42.021039 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.020882 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:38:42.149136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.149097 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-plugins-config-volume\") pod \"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358\" (UID: \"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358\") " Oct 29 22:38:42.149322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.149201 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-kube-api-access-h4dd6\" (UniqueName: \"kubernetes.io/projected/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-kube-api-access-h4dd6\") pod \"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358\" (UID: \"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358\") " Oct 29 22:38:42.149583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.149554 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-plugins-config-volume" pod "ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358" (UID: "ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:42.149892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.149869 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:42.151561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.151528 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-kube-api-access-h4dd6" pod "ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358" (UID: "ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358"). InnerVolumeSpecName "kube-api-access-h4dd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:42.183369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.183334 4952 generic.go:356] "Generic (PLEG): container finished" podID="ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358" containerID="8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e" exitCode=0 Oct 29 22:38:42.183527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.183376 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" Oct 29 22:38:42.183527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.183413 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" event={"ID":"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358","Type":"ContainerDied","Data":"8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e"} Oct 29 22:38:42.183527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.183469 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8" event={"ID":"ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358","Type":"ContainerDied","Data":"feef140a7abefa0433a68b93e763da5ea0113e2ccc95f6c9a4c3278e3a4127c9"} Oct 29 22:38:42.183527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.183492 4952 scope.go:122] "RemoveContainer" containerID="8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e" Oct 29 22:38:42.196008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.195989 4952 scope.go:122] "RemoveContainer" containerID="8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e" Oct 29 22:38:42.196302 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:42.196284 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e\": container with ID starting with 8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e not found: ID does not exist" containerID="8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e" Oct 29 22:38:42.196357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.196311 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e"} err="failed to get container status \"8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e\": rpc error: code = NotFound desc = could not find container \"8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e\": container with ID starting with 8ba059b0bbdbd228e6bd109824dd3404fc1460b0fbec9acde5f52a3949ac712e not found: ID does not exist" Oct 29 22:38:42.244579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.244487 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8"] Oct 29 22:38:42.251424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.251388 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4dd6\" (UniqueName: \"kubernetes.io/projected/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358-kube-api-access-h4dd6\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:42.251606 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.251505 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-797e0c-88dcbf68c-gwgj8"] Oct 29 22:38:42.649827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.649786 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46cb18bb-e422-46b6-bbea-0881c23bcd7f" path="/var/lib/kubelet/pods/46cb18bb-e422-46b6-bbea-0881c23bcd7f/volumes" Oct 29 22:38:42.650571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.650548 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358" path="/var/lib/kubelet/pods/ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358/volumes" Oct 29 22:38:42.989397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:42.989297 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:44.837235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:44.837198 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:45.533083 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:45.533033 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:38:46.561235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:46.556570 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:48.177461 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:48.177402 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:48.577643 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:48.577598 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:48.776021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:48.775978 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:48.845863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:48.845757 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv"] Oct 29 22:38:48.847062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:48.846480 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" containerID="cri-o://ef1202e90fe228756aeb958568f646f491ccbf4c56e1b3da57a1374a37a331d0" gracePeriod=30 Oct 29 22:38:48.859875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:48.859833 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch"] Oct 29 22:38:48.860341 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:48.860308 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" containerID="cri-o://e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d" gracePeriod=30 Oct 29 22:38:48.983930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:48.973189 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:50.035236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:50.035188 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:50.206892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:50.206856 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:51.776986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:51.776945 4952 patch_prober.go:29] interesting pod/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:38:51.777437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:51.777011 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:38:51.982680 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:51.982639 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:53.851929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:53.849722 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:54.205182 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.205154 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:38:54.312653 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.312607 4952 generic.go:356] "Generic (PLEG): container finished" podID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerID="ef1202e90fe228756aeb958568f646f491ccbf4c56e1b3da57a1374a37a331d0" exitCode=0 Oct 29 22:38:54.312856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.312680 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" event={"ID":"2fec7422-4663-4796-a686-a13c2d8c9d9f","Type":"ContainerDied","Data":"ef1202e90fe228756aeb958568f646f491ccbf4c56e1b3da57a1374a37a331d0"} Oct 29 22:38:54.312856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.312759 4952 scope.go:122] "RemoveContainer" containerID="023370a0306f54b1b66b96fd7602b766489c04ae2e90a0ca772558b55909954c" Oct 29 22:38:54.315827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.315512 4952 generic.go:356] "Generic (PLEG): container finished" podID="95f78edf-e3c7-4336-b114-312d2b342149" containerID="e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d" exitCode=0 Oct 29 22:38:54.315827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.315546 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" event={"ID":"95f78edf-e3c7-4336-b114-312d2b342149","Type":"ContainerDied","Data":"e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d"} Oct 29 22:38:54.315827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.315571 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" Oct 29 22:38:54.315827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.315595 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch" event={"ID":"95f78edf-e3c7-4336-b114-312d2b342149","Type":"ContainerDied","Data":"82dc41548797b776327d944d91893e6320eadab4d89a501a733ca3fefa28154e"} Oct 29 22:38:54.318242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.318214 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-kube-api-access-tjpkj\" (UniqueName: \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-kube-api-access-tjpkj\") pod \"95f78edf-e3c7-4336-b114-312d2b342149\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " Oct 29 22:38:54.318354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.318314 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f78edf-e3c7-4336-b114-312d2b342149-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f78edf-e3c7-4336-b114-312d2b342149-istiod-ca-cert\") pod \"95f78edf-e3c7-4336-b114-312d2b342149\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " Oct 29 22:38:54.318418 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.318366 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-envoy\") pod \"95f78edf-e3c7-4336-b114-312d2b342149\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " Oct 29 22:38:54.318476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.318427 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-istio-token\" (UniqueName: \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-istio-token\") pod \"95f78edf-e3c7-4336-b114-312d2b342149\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " Oct 29 22:38:54.318555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.318480 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-credential-socket\") pod \"95f78edf-e3c7-4336-b114-312d2b342149\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " Oct 29 22:38:54.318555 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.318517 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-certs\") pod \"95f78edf-e3c7-4336-b114-312d2b342149\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " Oct 29 22:38:54.318703 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.318684 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-socket\") pod \"95f78edf-e3c7-4336-b114-312d2b342149\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " Oct 29 22:38:54.318761 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.318748 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95f78edf-e3c7-4336-b114-312d2b342149-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f78edf-e3c7-4336-b114-312d2b342149-istio-podinfo\") pod \"95f78edf-e3c7-4336-b114-312d2b342149\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " Oct 29 22:38:54.318920 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.318886 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-data\") pod \"95f78edf-e3c7-4336-b114-312d2b342149\" (UID: \"95f78edf-e3c7-4336-b114-312d2b342149\") " Oct 29 22:38:54.320178 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.320147 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-data" pod "95f78edf-e3c7-4336-b114-312d2b342149" (UID: "95f78edf-e3c7-4336-b114-312d2b342149"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:54.320453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.320391 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-certs" pod "95f78edf-e3c7-4336-b114-312d2b342149" (UID: "95f78edf-e3c7-4336-b114-312d2b342149"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:54.320554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.320449 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-socket" pod "95f78edf-e3c7-4336-b114-312d2b342149" (UID: "95f78edf-e3c7-4336-b114-312d2b342149"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:54.320628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.320577 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-credential-socket" pod "95f78edf-e3c7-4336-b114-312d2b342149" (UID: "95f78edf-e3c7-4336-b114-312d2b342149"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:54.320954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.320888 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f78edf-e3c7-4336-b114-312d2b342149-istiod-ca-cert" pod "95f78edf-e3c7-4336-b114-312d2b342149" (UID: "95f78edf-e3c7-4336-b114-312d2b342149"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:54.339380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.339102 4952 scope.go:122] "RemoveContainer" containerID="e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d" Oct 29 22:38:54.339380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.339365 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95f78edf-e3c7-4336-b114-312d2b342149-istio-podinfo" pod "95f78edf-e3c7-4336-b114-312d2b342149" (UID: "95f78edf-e3c7-4336-b114-312d2b342149"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:54.339569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.339501 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-istio-token" pod "95f78edf-e3c7-4336-b114-312d2b342149" (UID: "95f78edf-e3c7-4336-b114-312d2b342149"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:54.344018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.342446 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-envoy" pod "95f78edf-e3c7-4336-b114-312d2b342149" (UID: "95f78edf-e3c7-4336-b114-312d2b342149"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:54.345556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.345533 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-kube-api-access-tjpkj" pod "95f78edf-e3c7-4336-b114-312d2b342149" (UID: "95f78edf-e3c7-4336-b114-312d2b342149"). InnerVolumeSpecName "kube-api-access-tjpkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:54.356170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.356145 4952 scope.go:122] "RemoveContainer" containerID="e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d" Oct 29 22:38:54.356529 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:38:54.356502 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d\": container with ID starting with e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d not found: ID does not exist" containerID="e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d" Oct 29 22:38:54.356630 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.356541 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d"} err="failed to get container status \"e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d\": rpc error: code = NotFound desc = could not find container \"e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d\": container with ID starting with e4860227ae638bc3e7a7174cc337293410c12db9b24cf9c29c81c4fa6d77da4d not found: ID does not exist" Oct 29 22:38:54.421977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.421939 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.421977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.421979 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.422154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.421994 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.422154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.422011 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.422154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.422026 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.422154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.422041 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f78edf-e3c7-4336-b114-312d2b342149-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.422154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.422056 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f78edf-e3c7-4336-b114-312d2b342149-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.422154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.422071 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjpkj\" (UniqueName: \"kubernetes.io/projected/95f78edf-e3c7-4336-b114-312d2b342149-kube-api-access-tjpkj\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.422154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.422087 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f78edf-e3c7-4336-b114-312d2b342149-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.588815 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.588787 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:38:54.716100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.716057 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch"] Oct 29 22:38:54.723928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.723477 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-nddch"] Oct 29 22:38:54.728123 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.728040 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-envoy\") pod \"2fec7422-4663-4796-a686-a13c2d8c9d9f\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " Oct 29 22:38:54.728123 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.728115 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-token\" (UniqueName: \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-token\") pod \"2fec7422-4663-4796-a686-a13c2d8c9d9f\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " Oct 29 22:38:54.728349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.728141 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fec7422-4663-4796-a686-a13c2d8c9d9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fec7422-4663-4796-a686-a13c2d8c9d9f-istiod-ca-cert\") pod \"2fec7422-4663-4796-a686-a13c2d8c9d9f\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " Oct 29 22:38:54.728349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.728204 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-certs\") pod \"2fec7422-4663-4796-a686-a13c2d8c9d9f\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " Oct 29 22:38:54.728349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.728276 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-data\") pod \"2fec7422-4663-4796-a686-a13c2d8c9d9f\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " Oct 29 22:38:54.728349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.728313 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-podinfo\") pod \"2fec7422-4663-4796-a686-a13c2d8c9d9f\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " Oct 29 22:38:54.728560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.728398 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-credential-socket\") pod \"2fec7422-4663-4796-a686-a13c2d8c9d9f\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " Oct 29 22:38:54.728560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.728482 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-kube-api-access-m2tl6\" (UniqueName: \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-kube-api-access-m2tl6\") pod \"2fec7422-4663-4796-a686-a13c2d8c9d9f\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " Oct 29 22:38:54.728560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.728499 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-socket\") pod \"2fec7422-4663-4796-a686-a13c2d8c9d9f\" (UID: \"2fec7422-4663-4796-a686-a13c2d8c9d9f\") " Oct 29 22:38:54.729612 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.729582 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-socket" pod "2fec7422-4663-4796-a686-a13c2d8c9d9f" (UID: "2fec7422-4663-4796-a686-a13c2d8c9d9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:54.729931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.729885 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fec7422-4663-4796-a686-a13c2d8c9d9f-istiod-ca-cert" pod "2fec7422-4663-4796-a686-a13c2d8c9d9f" (UID: "2fec7422-4663-4796-a686-a13c2d8c9d9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:54.730498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.730459 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-certs" pod "2fec7422-4663-4796-a686-a13c2d8c9d9f" (UID: "2fec7422-4663-4796-a686-a13c2d8c9d9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:54.730621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.730496 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-credential-socket" pod "2fec7422-4663-4796-a686-a13c2d8c9d9f" (UID: "2fec7422-4663-4796-a686-a13c2d8c9d9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:54.732248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.732225 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-data" pod "2fec7422-4663-4796-a686-a13c2d8c9d9f" (UID: "2fec7422-4663-4796-a686-a13c2d8c9d9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:54.732409 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.732386 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-podinfo" pod "2fec7422-4663-4796-a686-a13c2d8c9d9f" (UID: "2fec7422-4663-4796-a686-a13c2d8c9d9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:54.732875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.732854 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-token" pod "2fec7422-4663-4796-a686-a13c2d8c9d9f" (UID: "2fec7422-4663-4796-a686-a13c2d8c9d9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:54.734306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.734284 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-envoy" pod "2fec7422-4663-4796-a686-a13c2d8c9d9f" (UID: "2fec7422-4663-4796-a686-a13c2d8c9d9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:54.734737 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.734716 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-kube-api-access-m2tl6" pod "2fec7422-4663-4796-a686-a13c2d8c9d9f" (UID: "2fec7422-4663-4796-a686-a13c2d8c9d9f"). InnerVolumeSpecName "kube-api-access-m2tl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:54.831636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.831587 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.831636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.831622 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.831636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.831631 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.831636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.831641 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2tl6\" (UniqueName: \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-kube-api-access-m2tl6\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.832042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.831652 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.832042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.831667 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.832042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.831676 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fec7422-4663-4796-a686-a13c2d8c9d9f-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.832042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.831683 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fec7422-4663-4796-a686-a13c2d8c9d9f-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.832042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.831698 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fec7422-4663-4796-a686-a13c2d8c9d9f-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:38:54.998978 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:54.995107 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:38:55.328394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:55.328349 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" event={"ID":"2fec7422-4663-4796-a686-a13c2d8c9d9f","Type":"ContainerDied","Data":"a9b44d4f68a5ae6100d888f79b43fc96314642e99f8e8acc7963e6bafd192b6a"} Oct 29 22:38:55.328394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:55.328384 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv" Oct 29 22:38:55.328681 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:55.328420 4952 scope.go:122] "RemoveContainer" containerID="ef1202e90fe228756aeb958568f646f491ccbf4c56e1b3da57a1374a37a331d0" Oct 29 22:38:55.392595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:55.392547 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv"] Oct 29 22:38:55.397849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:55.397815 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-adfb9d-openshift-default-84884685b9-dsppv"] Oct 29 22:38:55.757982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:55.755420 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:56.817111 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:56.817074 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" path="/var/lib/kubelet/pods/2fec7422-4663-4796-a686-a13c2d8c9d9f/volumes" Oct 29 22:38:56.823958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:56.823921 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f78edf-e3c7-4336-b114-312d2b342149" path="/var/lib/kubelet/pods/95f78edf-e3c7-4336-b114-312d2b342149/volumes" Oct 29 22:38:57.856930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:57.846766 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:59.931929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:38:59.931094 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:00.170711 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:00.170670 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj"] Oct 29 22:39:00.207314 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:00.207226 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58"] Oct 29 22:39:00.380627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:00.380586 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:00.574888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:00.574568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:02.133522 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:02.133487 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:03.941257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:03.941171 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:04.113809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.113769 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:04.281475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.281374 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:04.455250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.452794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:04.815768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.815726 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v"] Oct 29 22:39:04.816890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.816858 4952 cpu_manager.go:414] "containerMap: removing container" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" Oct 29 22:39:04.816890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.816891 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" Oct 29 22:39:04.816890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.816925 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.816935 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.816950 4952 cpu_manager.go:414] "containerMap: removing container" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.816959 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.816982 4952 cpu_manager.go:414] "containerMap: removing container" podUID="46cb18bb-e422-46b6-bbea-0881c23bcd7f" containerName="payload-processing" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.816991 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46cb18bb-e422-46b6-bbea-0881c23bcd7f" containerName="payload-processing" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817018 4952 cpu_manager.go:414] "containerMap: removing container" podUID="21dd7134-a80f-4634-946d-1a23575d964b" containerName="payload-pre-processing" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817025 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dd7134-a80f-4634-946d-1a23575d964b" containerName="payload-pre-processing" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817045 4952 cpu_manager.go:414] "containerMap: removing container" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817053 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817069 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358" containerName="payload-pre-processing" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817077 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358" containerName="payload-pre-processing" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817089 4952 cpu_manager.go:414] "containerMap: removing container" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817098 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817117 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ffe5ea27-066f-4f88-9f21-e5e09d289018" containerName="payload-processing" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817124 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe5ea27-066f-4f88-9f21-e5e09d289018" containerName="payload-processing" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817140 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8b8892a7-74e6-422d-a0df-3d0ece183682" containerName="maas-api" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817147 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8892a7-74e6-422d-a0df-3d0ece183682" containerName="maas-api" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817157 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817165 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817188 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e7dd3058-c239-41cc-88dc-5d492be03792" containerName="istio-proxy" Oct 29 22:39:04.817190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817195 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7dd3058-c239-41cc-88dc-5d492be03792" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817209 4952 cpu_manager.go:414] "containerMap: removing container" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817216 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817232 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817239 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817264 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b9980446-05cb-46a2-961f-963f0313d496" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817271 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9980446-05cb-46a2-961f-963f0313d496" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817282 4952 cpu_manager.go:414] "containerMap: removing container" podUID="59b1d6bd-d0af-4db4-9d87-e50a8116a21e" containerName="maas-api" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817290 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b1d6bd-d0af-4db4-9d87-e50a8116a21e" containerName="maas-api" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817458 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="85eb5f26-c0eb-47a8-b260-ca2eed4ad3b2" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817471 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8d3b79-4d61-4a7b-bd2d-da8b40ef3358" containerName="payload-pre-processing" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817480 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817490 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe5ea27-066f-4f88-9f21-e5e09d289018" containerName="payload-processing" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817500 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8892a7-74e6-422d-a0df-3d0ece183682" containerName="maas-api" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817511 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817521 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7dd3058-c239-41cc-88dc-5d492be03792" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817529 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f78edf-e3c7-4336-b114-312d2b342149" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817542 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9980446-05cb-46a2-961f-963f0313d496" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817551 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b1d6bd-d0af-4db4-9d87-e50a8116a21e" containerName="maas-api" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817559 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fec7422-4663-4796-a686-a13c2d8c9d9f" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817571 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3543df-0b4d-4a83-ba52-ed177a1dfb93" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817579 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f735d4c-ebbc-4d37-b4f0-b7474f69ce8a" containerName="istio-proxy" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817589 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="46cb18bb-e422-46b6-bbea-0881c23bcd7f" containerName="payload-processing" Oct 29 22:39:04.818890 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.817601 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dd7134-a80f-4634-946d-1a23575d964b" containerName="payload-pre-processing" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.840197 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.885920 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-580af2-openshift-default-dockercfg-n7dzr\"" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.891041 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6862987c-7402-4134-a7a7-4a1663aed0de-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.891096 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.891136 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6862987c-7402-4134-a7a7-4a1663aed0de-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.891190 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.891227 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.891300 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmzbt\" (UniqueName: \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-kube-api-access-fmzbt\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.891435 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.891506 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.891538 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:04.899825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:04.893782 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v"] Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.094004 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6862987c-7402-4134-a7a7-4a1663aed0de-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.094063 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.094097 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6862987c-7402-4134-a7a7-4a1663aed0de-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.094143 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.094176 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.094240 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmzbt\" (UniqueName: \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-kube-api-access-fmzbt\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.094352 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.094411 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.094443 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.095106 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-credential-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.098927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.095823 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6862987c-7402-4134-a7a7-4a1663aed0de-istiod-ca-cert\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.136926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.136688 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:05.148966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.137895 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-socket\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.160974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.159303 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-data\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.160974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.159933 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-certs\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.339962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.324129 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" podUID="7f973a23-7f51-4459-b4db-c125fb274aae" containerName="payload-processing" containerID="cri-o://1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28" gracePeriod=25 Oct 29 22:39:05.339962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.327956 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" podUID="0f0f5be5-882d-4b1a-acc0-b967d5c7bce4" containerName="payload-pre-processing" containerID="cri-o://06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944" gracePeriod=25 Oct 29 22:39:05.450114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.450079 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:05.454944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.451701 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmzbt\" (UniqueName: \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-kube-api-access-fmzbt\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.478400 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.478364 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-envoy\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.500703 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.500607 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-istio-token\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.527664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.527590 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6862987c-7402-4134-a7a7-4a1663aed0de-istio-podinfo\") pod \"e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:05.810195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:05.810151 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:06.159951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:06.152390 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:06.211932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:06.210967 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v"] Oct 29 22:39:06.221633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:06.221591 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:39:06.221796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:06.221690 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:39:06.221796 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:06.221739 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:39:06.529898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:06.529798 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" event={"ID":"6862987c-7402-4134-a7a7-4a1663aed0de","Type":"ContainerStarted","Data":"62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39"} Oct 29 22:39:06.529898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:06.529861 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" event={"ID":"6862987c-7402-4134-a7a7-4a1663aed0de","Type":"ContainerStarted","Data":"b7295427a6f163f22cc89868e8b9a53905c9fdedf59c649f2386ba297c5ab93e"} Oct 29 22:39:06.584630 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:06.584564 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" podStartSLOduration=2.584545661 podStartE2EDuration="2.584545661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:06.579673083 +0000 UTC m=+3412.054261502" watchObservedRunningTime="2026-10-29 22:39:06.584545661 +0000 UTC m=+3412.059134079" Oct 29 22:39:06.810997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:06.810893 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:07.819009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:07.818565 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:07.835930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:07.834745 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:08.557981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:08.557941 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:08.559264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:08.559235 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:39:09.126253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:09.121320 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:09.298561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:09.298526 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:09.699324 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:09.699288 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:10.886048 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:10.886003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:10.999779 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:10.999742 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789"] Oct 29 22:39:11.000617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.000597 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3ce13d-da21-4593-8eba-603813119eea" containerName="istio-proxy" Oct 29 22:39:11.003322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.003307 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:39:11.030473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.030438 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789"] Oct 29 22:39:11.035854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.035817 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-580af2-dockercfg-bshwr\"" Oct 29 22:39:11.035854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.035817 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-580af2\"" Oct 29 22:39:11.066973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.066485 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:11.400931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.400116 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm"] Oct 29 22:39:11.438088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.405026 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" podUID="7f973a23-7f51-4459-b4db-c125fb274aae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:39:11.442971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.442899 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3b2165-83e9-4020-8159-f045ac452bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789\" (UID: \"ad3b2165-83e9-4020-8159-f045ac452bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:39:11.443209 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.443188 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssbzr\" (UniqueName: \"kubernetes.io/projected/ad3b2165-83e9-4020-8159-f045ac452bd0-kube-api-access-ssbzr\") pod \"payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789\" (UID: \"ad3b2165-83e9-4020-8159-f045ac452bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:39:11.485092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.485054 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:39:11.500647 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.500610 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm"] Oct 29 22:39:11.546659 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.546607 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76b81a-f826-46f6-95c7-27ed04ded082-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm\" (UID: \"6e76b81a-f826-46f6-95c7-27ed04ded082\") " pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:39:11.546876 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.546774 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3b2165-83e9-4020-8159-f045ac452bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789\" (UID: \"ad3b2165-83e9-4020-8159-f045ac452bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:39:11.546876 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.546801 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssbzr\" (UniqueName: \"kubernetes.io/projected/ad3b2165-83e9-4020-8159-f045ac452bd0-kube-api-access-ssbzr\") pod \"payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789\" (UID: \"ad3b2165-83e9-4020-8159-f045ac452bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:39:11.546876 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.546856 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdtb2\" (UniqueName: \"kubernetes.io/projected/6e76b81a-f826-46f6-95c7-27ed04ded082-kube-api-access-wdtb2\") pod \"payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm\" (UID: \"6e76b81a-f826-46f6-95c7-27ed04ded082\") " pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:39:11.547764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.547728 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3b2165-83e9-4020-8159-f045ac452bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789\" (UID: \"ad3b2165-83e9-4020-8159-f045ac452bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:39:11.572940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.572883 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssbzr\" (UniqueName: \"kubernetes.io/projected/ad3b2165-83e9-4020-8159-f045ac452bd0-kube-api-access-ssbzr\") pod \"payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789\" (UID: \"ad3b2165-83e9-4020-8159-f045ac452bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:39:11.590080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.590030 4952 generic.go:356] "Generic (PLEG): container finished" podID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerID="dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8" exitCode=0 Oct 29 22:39:11.590262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.590097 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" event={"ID":"85d2c654-5589-4747-986c-94ec4fd16ffd","Type":"ContainerDied","Data":"dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8"} Oct 29 22:39:11.590588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.590573 4952 scope.go:122] "RemoveContainer" containerID="dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8" Oct 29 22:39:11.592716 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.592688 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:39:11.592801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.592756 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:39:11.592801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.592791 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:39:11.617303 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.617278 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:39:11.648740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.648705 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76b81a-f826-46f6-95c7-27ed04ded082-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm\" (UID: \"6e76b81a-f826-46f6-95c7-27ed04ded082\") " pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:39:11.648896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.648871 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdtb2\" (UniqueName: \"kubernetes.io/projected/6e76b81a-f826-46f6-95c7-27ed04ded082-kube-api-access-wdtb2\") pod \"payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm\" (UID: \"6e76b81a-f826-46f6-95c7-27ed04ded082\") " pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:39:11.649501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.649459 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76b81a-f826-46f6-95c7-27ed04ded082-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm\" (UID: \"6e76b81a-f826-46f6-95c7-27ed04ded082\") " pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:39:11.677018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.676940 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdtb2\" (UniqueName: \"kubernetes.io/projected/6e76b81a-f826-46f6-95c7-27ed04ded082-kube-api-access-wdtb2\") pod \"payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm\" (UID: \"6e76b81a-f826-46f6-95c7-27ed04ded082\") " pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:39:11.736433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.735929 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:11.817137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.816854 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:39:11.955012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:11.954533 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789"] Oct 29 22:39:12.212799 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.212706 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm"] Oct 29 22:39:12.220222 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:39:12.220187 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e76b81a_f826_46f6_95c7_27ed04ded082.slice/crio-93ae1c1e712722ddc05802279bd326bbab76c6f9244be069a3e9acf28bf5349e WatchSource:0}: Error finding container 93ae1c1e712722ddc05802279bd326bbab76c6f9244be069a3e9acf28bf5349e: Status 404 returned error can't find the container with id 93ae1c1e712722ddc05802279bd326bbab76c6f9244be069a3e9acf28bf5349e Oct 29 22:39:12.254928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.254870 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" podUID="0f0f5be5-882d-4b1a-acc0-b967d5c7bce4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:39:12.602076 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.602039 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" event={"ID":"85d2c654-5589-4747-986c-94ec4fd16ffd","Type":"ContainerStarted","Data":"c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54"} Oct 29 22:39:12.604440 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.604407 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" event={"ID":"6e76b81a-f826-46f6-95c7-27ed04ded082","Type":"ContainerStarted","Data":"2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7"} Oct 29 22:39:12.604688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.604665 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" event={"ID":"6e76b81a-f826-46f6-95c7-27ed04ded082","Type":"ContainerStarted","Data":"93ae1c1e712722ddc05802279bd326bbab76c6f9244be069a3e9acf28bf5349e"} Oct 29 22:39:12.605083 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.605059 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:39:12.608845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.608819 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" event={"ID":"ad3b2165-83e9-4020-8159-f045ac452bd0","Type":"ContainerStarted","Data":"fe0951340f00dea00f2dd9d59a01089c5f23c2af3936c5fb19e94f268c9369ed"} Oct 29 22:39:12.608956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.608866 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" event={"ID":"ad3b2165-83e9-4020-8159-f045ac452bd0","Type":"ContainerStarted","Data":"2442214bcc7e1bcd84b4eef06a89873fd75ebb5f5baa7754149e898e5a2aa9cc"} Oct 29 22:39:12.609096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.609072 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:39:12.712243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.712181 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" podStartSLOduration=1.712164639 podStartE2EDuration="1.712164639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:12.709836124 +0000 UTC m=+3418.184424542" watchObservedRunningTime="2026-10-29 22:39:12.712164639 +0000 UTC m=+3418.186753060" Oct 29 22:39:12.770993 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:12.770929 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" podStartSLOduration=2.770897406 podStartE2EDuration="2.770897406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:12.765664484 +0000 UTC m=+3418.240252903" watchObservedRunningTime="2026-10-29 22:39:12.770897406 +0000 UTC m=+3418.245485827" Oct 29 22:39:13.006307 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:13.006158 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:13.257003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:13.251664 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:13.550986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:13.550284 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:39:13.553026 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:13.552993 4952 patch_prober.go:29] interesting pod/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 22:39:13.553815 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:13.553051 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 22:39:13.726928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:13.726639 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:14.152075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:14.152031 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:39:14.547493 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:14.547389 4952 patch_prober.go:29] interesting pod/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 22:39:14.548186 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:14.548147 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 22:39:15.555928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:15.554947 4952 patch_prober.go:29] interesting pod/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 22:39:15.555928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:15.555014 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 22:39:15.587924 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:39:15.587874 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:39:15.588184 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:15.588145 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:16.551989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:16.548485 4952 patch_prober.go:29] interesting pod/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 22:39:16.551989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:16.548580 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 22:39:16.659364 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:16.659324 4952 scope.go:122] "RemoveContainer" containerID="1b537c510231bde3cce31639dd7a1663adf9dd9dd0c04797fb48ceb093f731db" Oct 29 22:39:17.246757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:17.246710 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:17.550983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:17.550335 4952 patch_prober.go:29] interesting pod/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 29 22:39:17.550983 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:17.550401 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 29 22:39:18.553685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:18.553635 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:39:18.554889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:18.554861 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:39:18.948632 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:18.948588 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:20.548929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:20.547978 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:21.247065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:21.247022 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" podUID="7f973a23-7f51-4459-b4db-c125fb274aae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:39:22.154931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:22.153126 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:22.254606 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:22.254544 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" podUID="0f0f5be5-882d-4b1a-acc0-b967d5c7bce4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:39:22.832068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:22.832031 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp"] Oct 29 22:39:22.832300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:22.832260 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" containerID="cri-o://cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2" gracePeriod=30 Oct 29 22:39:23.658317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:23.658279 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:39:23.764216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:23.764177 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:24.676919 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:24.676872 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:39:24.873571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:24.873531 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:39:24.873771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:24.873596 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:39:25.368065 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.368025 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:25.719108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.719081 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:39:25.731840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.731805 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-plugins-config-volume\") pod \"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4\" (UID: \"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4\") " Oct 29 22:39:25.732244 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.732218 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-kube-api-access-vs94g\" (UniqueName: \"kubernetes.io/projected/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-kube-api-access-vs94g\") pod \"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4\" (UID: \"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4\") " Oct 29 22:39:25.732931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.732891 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-plugins-config-volume" pod "0f0f5be5-882d-4b1a-acc0-b967d5c7bce4" (UID: "0f0f5be5-882d-4b1a-acc0-b967d5c7bce4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:25.736357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.736330 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-kube-api-access-vs94g" pod "0f0f5be5-882d-4b1a-acc0-b967d5c7bce4" (UID: "0f0f5be5-882d-4b1a-acc0-b967d5c7bce4"). InnerVolumeSpecName "kube-api-access-vs94g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:25.770208 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.770176 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:39:25.809148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.809103 4952 generic.go:356] "Generic (PLEG): container finished" podID="7f973a23-7f51-4459-b4db-c125fb274aae" containerID="1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28" exitCode=0 Oct 29 22:39:25.809475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.809434 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" event={"ID":"7f973a23-7f51-4459-b4db-c125fb274aae","Type":"ContainerDied","Data":"1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28"} Oct 29 22:39:25.809564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.809513 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" event={"ID":"7f973a23-7f51-4459-b4db-c125fb274aae","Type":"ContainerDied","Data":"5fa855222b5618f33f63e52ec77c2bae51bba37b32c95abc173abfa10e28051b"} Oct 29 22:39:25.809630 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.809563 4952 scope.go:122] "RemoveContainer" containerID="1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28" Oct 29 22:39:25.809788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.809769 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj" Oct 29 22:39:25.812351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.812322 4952 generic.go:356] "Generic (PLEG): container finished" podID="0f0f5be5-882d-4b1a-acc0-b967d5c7bce4" containerID="06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944" exitCode=0 Oct 29 22:39:25.812476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.812439 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" event={"ID":"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4","Type":"ContainerDied","Data":"06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944"} Oct 29 22:39:25.812545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.812481 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" event={"ID":"0f0f5be5-882d-4b1a-acc0-b967d5c7bce4","Type":"ContainerDied","Data":"bcc21ec867935b97682928c7f2219ac921a74c377631cd9e05b1007b6caa0fc9"} Oct 29 22:39:25.812607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.812551 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58" Oct 29 22:39:25.827972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.827769 4952 scope.go:122] "RemoveContainer" containerID="1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28" Oct 29 22:39:25.828521 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:39:25.828490 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28\": container with ID starting with 1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28 not found: ID does not exist" containerID="1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28" Oct 29 22:39:25.828651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.828531 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28"} err="failed to get container status \"1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28\": rpc error: code = NotFound desc = could not find container \"1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28\": container with ID starting with 1ecf9b6481c9cb67d0f24e8f1d1a9747f86990caa403d88158bf178632912d28 not found: ID does not exist" Oct 29 22:39:25.828651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.828555 4952 scope.go:122] "RemoveContainer" containerID="06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944" Oct 29 22:39:25.835268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.835233 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f973a23-7f51-4459-b4db-c125fb274aae-kube-api-access-hmhfg\" (UniqueName: \"kubernetes.io/projected/7f973a23-7f51-4459-b4db-c125fb274aae-kube-api-access-hmhfg\") pod \"7f973a23-7f51-4459-b4db-c125fb274aae\" (UID: \"7f973a23-7f51-4459-b4db-c125fb274aae\") " Oct 29 22:39:25.835592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.835574 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f973a23-7f51-4459-b4db-c125fb274aae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f973a23-7f51-4459-b4db-c125fb274aae-plugins-config-volume\") pod \"7f973a23-7f51-4459-b4db-c125fb274aae\" (UID: \"7f973a23-7f51-4459-b4db-c125fb274aae\") " Oct 29 22:39:25.836201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.836179 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:25.836315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.836205 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs94g\" (UniqueName: \"kubernetes.io/projected/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4-kube-api-access-vs94g\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:25.840010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.839984 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f973a23-7f51-4459-b4db-c125fb274aae-plugins-config-volume" pod "7f973a23-7f51-4459-b4db-c125fb274aae" (UID: "7f973a23-7f51-4459-b4db-c125fb274aae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:25.845435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.845394 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f973a23-7f51-4459-b4db-c125fb274aae-kube-api-access-hmhfg" pod "7f973a23-7f51-4459-b4db-c125fb274aae" (UID: "7f973a23-7f51-4459-b4db-c125fb274aae"). InnerVolumeSpecName "kube-api-access-hmhfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:25.858251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.858226 4952 scope.go:122] "RemoveContainer" containerID="06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944" Oct 29 22:39:25.858640 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:39:25.858609 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944\": container with ID starting with 06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944 not found: ID does not exist" containerID="06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944" Oct 29 22:39:25.858758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.858652 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944"} err="failed to get container status \"06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944\": rpc error: code = NotFound desc = could not find container \"06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944\": container with ID starting with 06189105f89aff8261cd1d02d6350a5cb24eab5852883be89c87725d47a84944 not found: ID does not exist" Oct 29 22:39:25.895026 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.894950 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58"] Oct 29 22:39:25.899964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.899938 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05687d-748bb88d6c-5dm58"] Oct 29 22:39:25.937768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.937729 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmhfg\" (UniqueName: \"kubernetes.io/projected/7f973a23-7f51-4459-b4db-c125fb274aae-kube-api-access-hmhfg\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:25.938050 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:25.938023 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f973a23-7f51-4459-b4db-c125fb274aae-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:26.183955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:26.183849 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj"] Oct 29 22:39:26.194951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:26.194878 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-05687d-5f4cb78fd4-fz7jj"] Oct 29 22:39:26.647032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:26.646998 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0f5be5-882d-4b1a-acc0-b967d5c7bce4" path="/var/lib/kubelet/pods/0f0f5be5-882d-4b1a-acc0-b967d5c7bce4/volumes" Oct 29 22:39:26.647544 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:26.647526 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f973a23-7f51-4459-b4db-c125fb274aae" path="/var/lib/kubelet/pods/7f973a23-7f51-4459-b4db-c125fb274aae/volumes" Oct 29 22:39:27.044043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:27.044005 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:28.146145 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.146115 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 22:39:28.179235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179202 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-socket\") pod \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " Oct 29 22:39:28.179394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179351 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istiod-ca-cert\") pod \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " Oct 29 22:39:28.179394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179386 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-data\") pod \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " Oct 29 22:39:28.179474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179438 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-envoy\") pod \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " Oct 29 22:39:28.179538 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179507 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-socket" pod "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" (UID: "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:28.179599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179521 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-token\" (UniqueName: \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-token\") pod \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " Oct 29 22:39:28.179664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179616 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-kube-api-access-wxbvh\" (UniqueName: \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-kube-api-access-wxbvh\") pod \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " Oct 29 22:39:28.179822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179692 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-data" pod "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" (UID: "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:28.179822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179742 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-certs\") pod \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " Oct 29 22:39:28.179822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179771 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-podinfo\") pod \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " Oct 29 22:39:28.179822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179804 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-credential-socket\") pod \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\" (UID: \"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d\") " Oct 29 22:39:28.179822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.179802 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istiod-ca-cert" pod "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" (UID: "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:28.180215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.180188 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-certs" pod "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" (UID: "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:28.180340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.180282 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-credential-socket" pod "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" (UID: "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:28.180776 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.180753 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:28.180896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.180781 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:28.180896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.180798 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:28.180896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.180813 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:28.180896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.180827 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:28.181960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.181932 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-podinfo" pod "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" (UID: "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:28.182118 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.182101 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-envoy" pod "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" (UID: "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:28.182342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.182318 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-token" pod "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" (UID: "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:28.182546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.182519 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-kube-api-access-wxbvh" pod "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" (UID: "c9d14e43-1bb1-43cd-87d2-6a7f31c0523d"). InnerVolumeSpecName "kube-api-access-wxbvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:28.282369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.282337 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:28.282369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.282361 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:28.282369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.282371 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxbvh\" (UniqueName: \"kubernetes.io/projected/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-kube-api-access-wxbvh\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:28.282615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.282382 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:28.823987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.823885 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:28.885924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.884235 4952 generic.go:356] "Generic (PLEG): container finished" podID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerID="cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2" exitCode=0 Oct 29 22:39:28.885924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.884456 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" event={"ID":"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d","Type":"ContainerDied","Data":"cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2"} Oct 29 22:39:28.885924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.884494 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" event={"ID":"c9d14e43-1bb1-43cd-87d2-6a7f31c0523d","Type":"ContainerDied","Data":"6763362f66e99f470e7b1c8e737a1969b319a3a2fb3e6ffb9b0f7f5e84ec0e57"} Oct 29 22:39:28.885924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.884516 4952 scope.go:122] "RemoveContainer" containerID="cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2" Oct 29 22:39:28.885924 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.884631 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp" Oct 29 22:39:28.900552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.900523 4952 scope.go:122] "RemoveContainer" containerID="cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2" Oct 29 22:39:28.900854 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:39:28.900825 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2\": container with ID starting with cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2 not found: ID does not exist" containerID="cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2" Oct 29 22:39:28.900956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.900857 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2"} err="failed to get container status \"cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2\": rpc error: code = NotFound desc = could not find container \"cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2\": container with ID starting with cc6e75693e6f3ab65197cc7fed13e9ea59ccdb1281fd588ec067e218861bb4f2 not found: ID does not exist" Oct 29 22:39:28.947678 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.947590 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp"] Oct 29 22:39:28.960873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:28.960835 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-91520c-openshift-default-65967c64cf-r6lgp"] Oct 29 22:39:30.083367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:30.083333 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9"] Oct 29 22:39:30.083974 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:30.083546 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" podUID="d6b70588-0e21-4abb-bdec-e51dc4a54291" containerName="maas-api" containerID="cri-o://1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce" gracePeriod=30 Oct 29 22:39:30.201585 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:30.201541 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:30.843968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:30.778559 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:30.997987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:30.995226 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:30.999415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:30.999375 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" path="/var/lib/kubelet/pods/c9d14e43-1bb1-43cd-87d2-6a7f31c0523d/volumes" Oct 29 22:39:30.999970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:30.999945 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5"] Oct 29 22:39:31.000273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.000245 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" containerID="cri-o://d30cd9a97a780dc9e3060de21d27474e175382631905d2aa5308e51a967d4960" gracePeriod=30 Oct 29 22:39:31.001510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.001490 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:39:31.011042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.011006 4952 generic.go:356] "Generic (PLEG): container finished" podID="d6b70588-0e21-4abb-bdec-e51dc4a54291" containerID="1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce" exitCode=0 Oct 29 22:39:31.011185 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.011046 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" Oct 29 22:39:31.011185 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.011081 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" event={"ID":"d6b70588-0e21-4abb-bdec-e51dc4a54291","Type":"ContainerDied","Data":"1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce"} Oct 29 22:39:31.011185 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.011130 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9" event={"ID":"d6b70588-0e21-4abb-bdec-e51dc4a54291","Type":"ContainerDied","Data":"3a1313300b682b53cf5d4c82aa433cfcad4cd2f3cbc31e14789023ebe71fc275"} Oct 29 22:39:31.011185 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.011153 4952 scope.go:122] "RemoveContainer" containerID="1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce" Oct 29 22:39:31.023226 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.023197 4952 scope.go:122] "RemoveContainer" containerID="1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce" Oct 29 22:39:31.023706 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:39:31.023671 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce\": container with ID starting with 1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce not found: ID does not exist" containerID="1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce" Oct 29 22:39:31.023806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.023713 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce"} err="failed to get container status \"1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce\": rpc error: code = NotFound desc = could not find container \"1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce\": container with ID starting with 1e86819c0ac27e92e7c5c58e957a8d09b4c5a8e1be0a798cc557dc89a5ae00ce not found: ID does not exist" Oct 29 22:39:31.182136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.182050 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-metrics-certs\") pod \"d6b70588-0e21-4abb-bdec-e51dc4a54291\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " Oct 29 22:39:31.182617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.182377 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-maas-api-tls\") pod \"d6b70588-0e21-4abb-bdec-e51dc4a54291\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " Oct 29 22:39:31.182617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.182422 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b70588-0e21-4abb-bdec-e51dc4a54291-kube-api-access-zgm8t\" (UniqueName: \"kubernetes.io/projected/d6b70588-0e21-4abb-bdec-e51dc4a54291-kube-api-access-zgm8t\") pod \"d6b70588-0e21-4abb-bdec-e51dc4a54291\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " Oct 29 22:39:31.182617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.182468 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b70588-0e21-4abb-bdec-e51dc4a54291-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b70588-0e21-4abb-bdec-e51dc4a54291-openshift-service-ca\") pod \"d6b70588-0e21-4abb-bdec-e51dc4a54291\" (UID: \"d6b70588-0e21-4abb-bdec-e51dc4a54291\") " Oct 29 22:39:31.182969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.182935 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b70588-0e21-4abb-bdec-e51dc4a54291-openshift-service-ca" pod "d6b70588-0e21-4abb-bdec-e51dc4a54291" (UID: "d6b70588-0e21-4abb-bdec-e51dc4a54291"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:31.183302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.183276 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b70588-0e21-4abb-bdec-e51dc4a54291-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:31.184628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.184600 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-metrics-certs" pod "d6b70588-0e21-4abb-bdec-e51dc4a54291" (UID: "d6b70588-0e21-4abb-bdec-e51dc4a54291"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:31.184831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.184804 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b70588-0e21-4abb-bdec-e51dc4a54291-kube-api-access-zgm8t" pod "d6b70588-0e21-4abb-bdec-e51dc4a54291" (UID: "d6b70588-0e21-4abb-bdec-e51dc4a54291"). InnerVolumeSpecName "kube-api-access-zgm8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:31.185061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.185042 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-maas-api-tls" pod "d6b70588-0e21-4abb-bdec-e51dc4a54291" (UID: "d6b70588-0e21-4abb-bdec-e51dc4a54291"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:39:31.197003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.196975 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:31.286103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.285020 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:31.286103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.285056 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgm8t\" (UniqueName: \"kubernetes.io/projected/d6b70588-0e21-4abb-bdec-e51dc4a54291-kube-api-access-zgm8t\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:31.286103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.285069 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6b70588-0e21-4abb-bdec-e51dc4a54291-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:31.368400 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.367865 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9"] Oct 29 22:39:31.374289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.374242 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26abe8-77b7dd958c-wzvp9"] Oct 29 22:39:31.533975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:31.533873 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:32.649608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:32.649575 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b70588-0e21-4abb-bdec-e51dc4a54291" path="/var/lib/kubelet/pods/d6b70588-0e21-4abb-bdec-e51dc4a54291/volumes" Oct 29 22:39:33.103006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:33.080852 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:34.493874 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:34.493828 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:35.249435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:35.249390 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:35.415838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:35.415797 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:35.584975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:35.584929 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:35.917208 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:35.917105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:36.163544 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.163504 4952 generic.go:356] "Generic (PLEG): container finished" podID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerID="d30cd9a97a780dc9e3060de21d27474e175382631905d2aa5308e51a967d4960" exitCode=0 Oct 29 22:39:36.163741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.163567 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" event={"ID":"b01b1e90-e2d0-4547-a491-f91d584b6253","Type":"ContainerDied","Data":"d30cd9a97a780dc9e3060de21d27474e175382631905d2aa5308e51a967d4960"} Oct 29 22:39:36.317685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.317650 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:39:36.385523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.385484 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-kube-api-access-nlrhb\" (UniqueName: \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-kube-api-access-nlrhb\") pod \"b01b1e90-e2d0-4547-a491-f91d584b6253\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " Oct 29 22:39:36.385715 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.385567 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-envoy\") pod \"b01b1e90-e2d0-4547-a491-f91d584b6253\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " Oct 29 22:39:36.385715 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.385601 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-podinfo\") pod \"b01b1e90-e2d0-4547-a491-f91d584b6253\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " Oct 29 22:39:36.385715 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.385637 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-socket\") pod \"b01b1e90-e2d0-4547-a491-f91d584b6253\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " Oct 29 22:39:36.385715 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.385688 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01b1e90-e2d0-4547-a491-f91d584b6253-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01b1e90-e2d0-4547-a491-f91d584b6253-istiod-ca-cert\") pod \"b01b1e90-e2d0-4547-a491-f91d584b6253\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " Oct 29 22:39:36.386038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.385835 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-certs\") pod \"b01b1e90-e2d0-4547-a491-f91d584b6253\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " Oct 29 22:39:36.386038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.385892 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-token\" (UniqueName: \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-token\") pod \"b01b1e90-e2d0-4547-a491-f91d584b6253\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " Oct 29 22:39:36.386038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.386027 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-credential-socket\") pod \"b01b1e90-e2d0-4547-a491-f91d584b6253\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " Oct 29 22:39:36.386215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.386098 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-data\") pod \"b01b1e90-e2d0-4547-a491-f91d584b6253\" (UID: \"b01b1e90-e2d0-4547-a491-f91d584b6253\") " Oct 29 22:39:36.388096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.388051 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-data" pod "b01b1e90-e2d0-4547-a491-f91d584b6253" (UID: "b01b1e90-e2d0-4547-a491-f91d584b6253"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:36.391660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.391590 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-socket" pod "b01b1e90-e2d0-4547-a491-f91d584b6253" (UID: "b01b1e90-e2d0-4547-a491-f91d584b6253"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:36.391660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.391616 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-credential-socket" pod "b01b1e90-e2d0-4547-a491-f91d584b6253" (UID: "b01b1e90-e2d0-4547-a491-f91d584b6253"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:36.392106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.391967 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-certs" pod "b01b1e90-e2d0-4547-a491-f91d584b6253" (UID: "b01b1e90-e2d0-4547-a491-f91d584b6253"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:36.398360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.393388 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-kube-api-access-nlrhb" pod "b01b1e90-e2d0-4547-a491-f91d584b6253" (UID: "b01b1e90-e2d0-4547-a491-f91d584b6253"). InnerVolumeSpecName "kube-api-access-nlrhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:36.398634 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.398597 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-envoy" pod "b01b1e90-e2d0-4547-a491-f91d584b6253" (UID: "b01b1e90-e2d0-4547-a491-f91d584b6253"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:36.398730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.398662 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-podinfo" pod "b01b1e90-e2d0-4547-a491-f91d584b6253" (UID: "b01b1e90-e2d0-4547-a491-f91d584b6253"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:36.399252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.399225 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01b1e90-e2d0-4547-a491-f91d584b6253-istiod-ca-cert" pod "b01b1e90-e2d0-4547-a491-f91d584b6253" (UID: "b01b1e90-e2d0-4547-a491-f91d584b6253"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:36.403339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.403310 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-token" pod "b01b1e90-e2d0-4547-a491-f91d584b6253" (UID: "b01b1e90-e2d0-4547-a491-f91d584b6253"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:36.490867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.490772 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlrhb\" (UniqueName: \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-kube-api-access-nlrhb\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.490867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.490813 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.490867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.490831 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.490867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.490845 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.490867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.490861 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01b1e90-e2d0-4547-a491-f91d584b6253-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.491333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.490879 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.491333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.490894 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.491333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.490924 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:36.491333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:36.490938 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01b1e90-e2d0-4547-a491-f91d584b6253-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:37.181435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:37.181392 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" event={"ID":"b01b1e90-e2d0-4547-a491-f91d584b6253","Type":"ContainerDied","Data":"eebae58f7b7ccf3acb6c135d5e6e854b6c90c5d17de939871644214712d3184f"} Oct 29 22:39:37.182002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:37.181473 4952 scope.go:122] "RemoveContainer" containerID="d30cd9a97a780dc9e3060de21d27474e175382631905d2aa5308e51a967d4960" Oct 29 22:39:37.182002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:37.181627 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5" Oct 29 22:39:37.258427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:37.258389 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5"] Oct 29 22:39:37.268842 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:37.268809 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xfcv5"] Oct 29 22:39:37.394931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:37.393456 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:38.647586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:38.647546 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" path="/var/lib/kubelet/pods/b01b1e90-e2d0-4547-a491-f91d584b6253/volumes" Oct 29 22:39:39.181926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:39.176537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:39.417781 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:39.417738 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:39.590039 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:39.589822 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:39.721376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:39.721338 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt"] Oct 29 22:39:39.722195 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:39.722156 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" containerID="cri-o://ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a" gracePeriod=30 Oct 29 22:39:40.907194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:40.907144 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:41.121127 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:41.120995 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:41.824386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:41.824318 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:41.999117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:41.999079 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:42.192580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.192487 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm"] Oct 29 22:39:42.193296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193272 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" Oct 29 22:39:42.193296 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193296 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" Oct 29 22:39:42.193498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193322 4952 cpu_manager.go:414] "containerMap: removing container" podUID="0f0f5be5-882d-4b1a-acc0-b967d5c7bce4" containerName="payload-pre-processing" Oct 29 22:39:42.193498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193331 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0f5be5-882d-4b1a-acc0-b967d5c7bce4" containerName="payload-pre-processing" Oct 29 22:39:42.193498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193353 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d6b70588-0e21-4abb-bdec-e51dc4a54291" containerName="maas-api" Oct 29 22:39:42.193498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193360 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b70588-0e21-4abb-bdec-e51dc4a54291" containerName="maas-api" Oct 29 22:39:42.193498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193380 4952 cpu_manager.go:414] "containerMap: removing container" podUID="7f973a23-7f51-4459-b4db-c125fb274aae" containerName="payload-processing" Oct 29 22:39:42.193498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193388 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f973a23-7f51-4459-b4db-c125fb274aae" containerName="payload-processing" Oct 29 22:39:42.193498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193407 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" Oct 29 22:39:42.193498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193414 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" Oct 29 22:39:42.194426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193654 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f973a23-7f51-4459-b4db-c125fb274aae" containerName="payload-processing" Oct 29 22:39:42.194426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193671 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d14e43-1bb1-43cd-87d2-6a7f31c0523d" containerName="istio-proxy" Oct 29 22:39:42.194426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193692 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01b1e90-e2d0-4547-a491-f91d584b6253" containerName="istio-proxy" Oct 29 22:39:42.194426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193703 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0f5be5-882d-4b1a-acc0-b967d5c7bce4" containerName="payload-pre-processing" Oct 29 22:39:42.194426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.193715 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b70588-0e21-4abb-bdec-e51dc4a54291" containerName="maas-api" Oct 29 22:39:42.197434 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.197405 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.226404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.226356 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm"] Oct 29 22:39:42.229516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.229488 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d09411\"/\"default-dockercfg-j4j9g\"" Oct 29 22:39:42.229665 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.229642 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d09411\"/\"test-model-d09411-kserve-self-signed-certs\"" Oct 29 22:39:42.229763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.229746 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d09411\"/\"openshift-service-ca.crt\"" Oct 29 22:39:42.229835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.229826 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d09411\"/\"kube-root-ca.crt\"" Oct 29 22:39:42.296438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.296402 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-tmp-dir\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.296676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.296456 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-dshm\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.296676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.296572 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-model-cache\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.296825 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.296672 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-home\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.296888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.296858 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a61920-3f20-4e73-84f8-dbafb64381f1-tls-certs\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.296982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.296934 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k24kc\" (UniqueName: \"kubernetes.io/projected/72a61920-3f20-4e73-84f8-dbafb64381f1-kube-api-access-k24kc\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.303559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.303523 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:42.398143 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.398104 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-tmp-dir\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.398361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.398150 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-dshm\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.398361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.398183 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-model-cache\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.398361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.398218 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-home\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.398560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.398389 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a61920-3f20-4e73-84f8-dbafb64381f1-tls-certs\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.398560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.398426 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k24kc\" (UniqueName: \"kubernetes.io/projected/72a61920-3f20-4e73-84f8-dbafb64381f1-kube-api-access-k24kc\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.398763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.398721 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-tmp-dir\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.398843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.398764 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-home\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.398969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.398940 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-model-cache\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.400823 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.400795 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-dshm\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.401299 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.401277 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a61920-3f20-4e73-84f8-dbafb64381f1-tls-certs\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.458991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.451568 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k24kc\" (UniqueName: \"kubernetes.io/projected/72a61920-3f20-4e73-84f8-dbafb64381f1-kube-api-access-k24kc\") pod \"test-model-d09411-kserve-d5c87d9df-vjltm\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.511138 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.509436 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:42.527224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.526850 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:42.944509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:42.944467 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm"] Oct 29 22:39:42.948941 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:39:42.948872 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72a61920_3f20_4e73_84f8_dbafb64381f1.slice/crio-59fa0b7f53af483dcdb350c9d21e6b326443abaf832c7522653e9dd3dd03bb0f WatchSource:0}: Error finding container 59fa0b7f53af483dcdb350c9d21e6b326443abaf832c7522653e9dd3dd03bb0f: Status 404 returned error can't find the container with id 59fa0b7f53af483dcdb350c9d21e6b326443abaf832c7522653e9dd3dd03bb0f Oct 29 22:39:43.290608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:43.290508 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" event={"ID":"72a61920-3f20-4e73-84f8-dbafb64381f1","Type":"ContainerStarted","Data":"b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf"} Oct 29 22:39:43.290608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:43.290556 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" event={"ID":"72a61920-3f20-4e73-84f8-dbafb64381f1","Type":"ContainerStarted","Data":"59fa0b7f53af483dcdb350c9d21e6b326443abaf832c7522653e9dd3dd03bb0f"} Oct 29 22:39:43.336799 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:43.336747 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" podStartSLOduration=1.336733088 podStartE2EDuration="1.336733088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:43.33476821 +0000 UTC m=+3448.809356629" watchObservedRunningTime="2026-10-29 22:39:43.336733088 +0000 UTC m=+3448.811321522" Oct 29 22:39:43.527829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:43.527790 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:43.725385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:43.725341 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:43.913558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:43.913517 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:44.079611 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:44.079560 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:44.257923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:44.256942 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:45.035146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.035113 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:39:45.196394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196345 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-socket\") pod \"9f904684-fe8c-4e96-befb-229dbf30522f\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " Oct 29 22:39:45.196608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196437 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-certs\") pod \"9f904684-fe8c-4e96-befb-229dbf30522f\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " Oct 29 22:39:45.196608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196474 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-credential-socket\") pod \"9f904684-fe8c-4e96-befb-229dbf30522f\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " Oct 29 22:39:45.196608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196498 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-data\") pod \"9f904684-fe8c-4e96-befb-229dbf30522f\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " Oct 29 22:39:45.196608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196590 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f904684-fe8c-4e96-befb-229dbf30522f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f904684-fe8c-4e96-befb-229dbf30522f-istiod-ca-cert\") pod \"9f904684-fe8c-4e96-befb-229dbf30522f\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " Oct 29 22:39:45.196848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196673 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-kube-api-access-p9x7x\" (UniqueName: \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-kube-api-access-p9x7x\") pod \"9f904684-fe8c-4e96-befb-229dbf30522f\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " Oct 29 22:39:45.196848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196675 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-socket" pod "9f904684-fe8c-4e96-befb-229dbf30522f" (UID: "9f904684-fe8c-4e96-befb-229dbf30522f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:45.196848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196694 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-certs" pod "9f904684-fe8c-4e96-befb-229dbf30522f" (UID: "9f904684-fe8c-4e96-befb-229dbf30522f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:45.196848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196769 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-credential-socket" pod "9f904684-fe8c-4e96-befb-229dbf30522f" (UID: "9f904684-fe8c-4e96-befb-229dbf30522f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:45.196848 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196773 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f904684-fe8c-4e96-befb-229dbf30522f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f904684-fe8c-4e96-befb-229dbf30522f-istio-podinfo\") pod \"9f904684-fe8c-4e96-befb-229dbf30522f\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " Oct 29 22:39:45.197159 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196857 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-data" pod "9f904684-fe8c-4e96-befb-229dbf30522f" (UID: "9f904684-fe8c-4e96-befb-229dbf30522f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:45.197159 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196879 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-envoy\") pod \"9f904684-fe8c-4e96-befb-229dbf30522f\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " Oct 29 22:39:45.197159 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.196953 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-istio-token\" (UniqueName: \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-istio-token\") pod \"9f904684-fe8c-4e96-befb-229dbf30522f\" (UID: \"9f904684-fe8c-4e96-befb-229dbf30522f\") " Oct 29 22:39:45.197159 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.197111 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f904684-fe8c-4e96-befb-229dbf30522f-istiod-ca-cert" pod "9f904684-fe8c-4e96-befb-229dbf30522f" (UID: "9f904684-fe8c-4e96-befb-229dbf30522f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:45.197691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.197668 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.197691 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.197690 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.197889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.197699 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.197889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.197708 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.197889 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.197716 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f904684-fe8c-4e96-befb-229dbf30522f-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.199163 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.199132 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f904684-fe8c-4e96-befb-229dbf30522f-istio-podinfo" pod "9f904684-fe8c-4e96-befb-229dbf30522f" (UID: "9f904684-fe8c-4e96-befb-229dbf30522f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:45.199481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.199454 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-istio-token" pod "9f904684-fe8c-4e96-befb-229dbf30522f" (UID: "9f904684-fe8c-4e96-befb-229dbf30522f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:45.199481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.199470 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-envoy" pod "9f904684-fe8c-4e96-befb-229dbf30522f" (UID: "9f904684-fe8c-4e96-befb-229dbf30522f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:45.199641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.199506 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-kube-api-access-p9x7x" pod "9f904684-fe8c-4e96-befb-229dbf30522f" (UID: "9f904684-fe8c-4e96-befb-229dbf30522f"). InnerVolumeSpecName "kube-api-access-p9x7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:45.299354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.299251 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9x7x\" (UniqueName: \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-kube-api-access-p9x7x\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.299354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.299294 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f904684-fe8c-4e96-befb-229dbf30522f-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.299354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.299309 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f904684-fe8c-4e96-befb-229dbf30522f-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.299354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.299320 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f904684-fe8c-4e96-befb-229dbf30522f-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:45.373742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.373698 4952 generic.go:356] "Generic (PLEG): container finished" podID="9f904684-fe8c-4e96-befb-229dbf30522f" containerID="ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a" exitCode=0 Oct 29 22:39:45.373972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.373756 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" Oct 29 22:39:45.373972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.373780 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" event={"ID":"9f904684-fe8c-4e96-befb-229dbf30522f","Type":"ContainerDied","Data":"ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a"} Oct 29 22:39:45.373972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.373849 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt" event={"ID":"9f904684-fe8c-4e96-befb-229dbf30522f","Type":"ContainerDied","Data":"13d613454dd90e1d7210aaf712c52ee1649bef1c9d85e5070dda7fa7bede612d"} Oct 29 22:39:45.373972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.373879 4952 scope.go:122] "RemoveContainer" containerID="ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a" Oct 29 22:39:45.388023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.388000 4952 scope.go:122] "RemoveContainer" containerID="b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7" Oct 29 22:39:45.401345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.401321 4952 scope.go:122] "RemoveContainer" containerID="ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a" Oct 29 22:39:45.401750 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:39:45.401728 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a\": container with ID starting with ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a not found: ID does not exist" containerID="ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a" Oct 29 22:39:45.401842 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.401762 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a"} err="failed to get container status \"ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a\": rpc error: code = NotFound desc = could not find container \"ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a\": container with ID starting with ceaad8839c9b93c6dd8327252f84052c7abb72eb16b73afce1bc0a4426c4336a not found: ID does not exist" Oct 29 22:39:45.401842 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.401788 4952 scope.go:122] "RemoveContainer" containerID="b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7" Oct 29 22:39:45.402101 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:39:45.402080 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7\": container with ID starting with b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7 not found: ID does not exist" containerID="b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7" Oct 29 22:39:45.402168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.402112 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7"} err="failed to get container status \"b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7\": rpc error: code = NotFound desc = could not find container \"b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7\": container with ID starting with b2ec39668b4a4f10c911274e20009e5fd1562f269f4c19de8b1f099b8fc982e7 not found: ID does not exist" Oct 29 22:39:45.436316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.436281 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt"] Oct 29 22:39:45.449281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.449242 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-26abe8-openshift-default-77569d6877-2x2dt"] Oct 29 22:39:45.533957 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:39:45.533885 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:39:45.825790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:45.825751 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:46.652524 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:46.652486 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" path="/var/lib/kubelet/pods/9f904684-fe8c-4e96-befb-229dbf30522f/volumes" Oct 29 22:39:46.798023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:46.797981 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:46.971714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:46.971599 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:47.148261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:47.148224 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:47.470801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:47.470736 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:48.731283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:48.731237 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:49.696362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:49.696317 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:49.873526 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:49.873488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:49.944785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:49.944736 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn"] Oct 29 22:39:49.945118 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:49.945054 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerName="istio-proxy" containerID="cri-o://bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe" gracePeriod=30 Oct 29 22:39:50.099935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:50.096254 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:50.570641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:50.570540 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:51.614595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:51.614558 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:51.811819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:51.811756 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:52.528549 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:52.528510 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:52.543136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:52.543101 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:52.654019 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:52.647214 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:52.727256 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:52.727220 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:39:54.593422 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:54.593324 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:54.785616 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:54.785576 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:55.076398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.076354 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:55.516317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.516277 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:39:55.663940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.663825 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-data\") pod \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " Oct 29 22:39:55.663940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.663886 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-certs\") pod \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.663957 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-envoy\") pod \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664022 4952 generic.go:356] "Generic (PLEG): container finished" podID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerID="bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe" exitCode=0 Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664059 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-credential-socket\") pod \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664096 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" event={"ID":"e6add6ee-6c29-4d7e-94f1-ddb3698c1244","Type":"ContainerDied","Data":"bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe"} Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664150 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" event={"ID":"e6add6ee-6c29-4d7e-94f1-ddb3698c1244","Type":"ContainerDied","Data":"3e473fe6f6f2f10cae8cadc62c4bca69ca44a38e4bcce0564d68c3ef42b9d079"} Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664182 4952 scope.go:122] "RemoveContainer" containerID="bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe" Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664260 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-data" pod "e6add6ee-6c29-4d7e-94f1-ddb3698c1244" (UID: "e6add6ee-6c29-4d7e-94f1-ddb3698c1244"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664277 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-certs" pod "e6add6ee-6c29-4d7e-94f1-ddb3698c1244" (UID: "e6add6ee-6c29-4d7e-94f1-ddb3698c1244"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664102 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-kube-api-access-qg5wv\" (UniqueName: \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-kube-api-access-qg5wv\") pod \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664355 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn" Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664361 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-socket\") pod \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664408 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-token\" (UniqueName: \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-token\") pod \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " Oct 29 22:39:55.664512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664449 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-podinfo\") pod \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " Oct 29 22:39:55.665999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.664572 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istiod-ca-cert\") pod \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\" (UID: \"e6add6ee-6c29-4d7e-94f1-ddb3698c1244\") " Oct 29 22:39:55.665999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.665314 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-credential-socket" pod "e6add6ee-6c29-4d7e-94f1-ddb3698c1244" (UID: "e6add6ee-6c29-4d7e-94f1-ddb3698c1244"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.665999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.665409 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.665999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.665433 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.665999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.665739 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-socket" pod "e6add6ee-6c29-4d7e-94f1-ddb3698c1244" (UID: "e6add6ee-6c29-4d7e-94f1-ddb3698c1244"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.665999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.665746 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istiod-ca-cert" pod "e6add6ee-6c29-4d7e-94f1-ddb3698c1244" (UID: "e6add6ee-6c29-4d7e-94f1-ddb3698c1244"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:55.667338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.667315 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-envoy" pod "e6add6ee-6c29-4d7e-94f1-ddb3698c1244" (UID: "e6add6ee-6c29-4d7e-94f1-ddb3698c1244"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.667719 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.667694 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-kube-api-access-qg5wv" pod "e6add6ee-6c29-4d7e-94f1-ddb3698c1244" (UID: "e6add6ee-6c29-4d7e-94f1-ddb3698c1244"). InnerVolumeSpecName "kube-api-access-qg5wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.668134 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.668115 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-token" pod "e6add6ee-6c29-4d7e-94f1-ddb3698c1244" (UID: "e6add6ee-6c29-4d7e-94f1-ddb3698c1244"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.669692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.669661 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-podinfo" pod "e6add6ee-6c29-4d7e-94f1-ddb3698c1244" (UID: "e6add6ee-6c29-4d7e-94f1-ddb3698c1244"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:55.702185 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.702111 4952 scope.go:122] "RemoveContainer" containerID="bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe" Oct 29 22:39:55.702515 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:39:55.702485 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe\": container with ID starting with bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe not found: ID does not exist" containerID="bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe" Oct 29 22:39:55.702617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.702527 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe"} err="failed to get container status \"bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe\": rpc error: code = NotFound desc = could not find container \"bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe\": container with ID starting with bbcb3591f1d2884f2662fc4139eccbb1fb6ea82a22bcab1fb4ef0b9e49c77dfe not found: ID does not exist" Oct 29 22:39:55.767334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.767292 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.767334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.767332 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.767571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.767346 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg5wv\" (UniqueName: \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-kube-api-access-qg5wv\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.767571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.767362 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.767571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.767377 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.767571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.767388 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.767571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:55.767398 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6add6ee-6c29-4d7e-94f1-ddb3698c1244-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:39:56.023415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:56.023374 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn"] Oct 29 22:39:56.032775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:56.032709 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05687d-openshift-default-759ffdf875-d5jxn"] Oct 29 22:39:56.555583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:56.554138 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:56.665925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:56.651397 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" path="/var/lib/kubelet/pods/e6add6ee-6c29-4d7e-94f1-ddb3698c1244/volumes" Oct 29 22:39:56.745009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:56.744946 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:56.960932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:56.955467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:58.673071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:39:58.671704 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:00.315939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:00.315090 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:01.218726 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:01.218544 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:01.394896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:01.394853 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:02.418751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:02.418709 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:03.288096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:03.288061 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:04.012406 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:04.012360 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:04.419671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:04.419626 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:05.414981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:05.414881 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:06.284884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:06.284848 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:06.472821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:06.472775 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:06.927920 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:06.927862 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:07.893093 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:07.893058 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5"] Oct 29 22:40:07.893934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:07.893857 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" podUID="4f6c96a8-b18d-4f10-8147-34df971af0c3" containerName="maas-api" containerID="cri-o://67914fcb09125fdd25fd723adbf5ef9f166e604bb58a441ccb6cd0b4ba9cd879" gracePeriod=30 Oct 29 22:40:08.000976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.000899 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:08.055831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.055794 4952 generic.go:356] "Generic (PLEG): container finished" podID="4f6c96a8-b18d-4f10-8147-34df971af0c3" containerID="67914fcb09125fdd25fd723adbf5ef9f166e604bb58a441ccb6cd0b4ba9cd879" exitCode=0 Oct 29 22:40:08.056101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.055961 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" event={"ID":"4f6c96a8-b18d-4f10-8147-34df971af0c3","Type":"ContainerDied","Data":"67914fcb09125fdd25fd723adbf5ef9f166e604bb58a441ccb6cd0b4ba9cd879"} Oct 29 22:40:08.631497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.631464 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:40:08.666243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.664460 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-metrics-certs\") pod \"4f6c96a8-b18d-4f10-8147-34df971af0c3\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " Oct 29 22:40:08.666243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.664695 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f6c96a8-b18d-4f10-8147-34df971af0c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f6c96a8-b18d-4f10-8147-34df971af0c3-openshift-service-ca\") pod \"4f6c96a8-b18d-4f10-8147-34df971af0c3\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " Oct 29 22:40:08.666243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.664765 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6c96a8-b18d-4f10-8147-34df971af0c3-kube-api-access-ffzmt\" (UniqueName: \"kubernetes.io/projected/4f6c96a8-b18d-4f10-8147-34df971af0c3-kube-api-access-ffzmt\") pod \"4f6c96a8-b18d-4f10-8147-34df971af0c3\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " Oct 29 22:40:08.666243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.664839 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-maas-api-tls\") pod \"4f6c96a8-b18d-4f10-8147-34df971af0c3\" (UID: \"4f6c96a8-b18d-4f10-8147-34df971af0c3\") " Oct 29 22:40:08.667107 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.667072 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6c96a8-b18d-4f10-8147-34df971af0c3-openshift-service-ca" pod "4f6c96a8-b18d-4f10-8147-34df971af0c3" (UID: "4f6c96a8-b18d-4f10-8147-34df971af0c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:08.680225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.680149 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6c96a8-b18d-4f10-8147-34df971af0c3-kube-api-access-ffzmt" pod "4f6c96a8-b18d-4f10-8147-34df971af0c3" (UID: "4f6c96a8-b18d-4f10-8147-34df971af0c3"). InnerVolumeSpecName "kube-api-access-ffzmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:08.686564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.686467 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f6c96a8-b18d-4f10-8147-34df971af0c3-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:08.686564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.686502 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffzmt\" (UniqueName: \"kubernetes.io/projected/4f6c96a8-b18d-4f10-8147-34df971af0c3-kube-api-access-ffzmt\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:08.694172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.694137 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-metrics-certs" pod "4f6c96a8-b18d-4f10-8147-34df971af0c3" (UID: "4f6c96a8-b18d-4f10-8147-34df971af0c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:08.695303 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.695271 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-maas-api-tls" pod "4f6c96a8-b18d-4f10-8147-34df971af0c3" (UID: "4f6c96a8-b18d-4f10-8147-34df971af0c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:08.788137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.788093 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:08.788137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.788128 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f6c96a8-b18d-4f10-8147-34df971af0c3-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:08.832465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:08.832402 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:09.068403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:09.068367 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" Oct 29 22:40:09.068403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:09.068392 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5" event={"ID":"4f6c96a8-b18d-4f10-8147-34df971af0c3","Type":"ContainerDied","Data":"b43b300daae8fd7ca62bcfb70045d8f2ebed459a8ca63c7caccaa5f673906844"} Oct 29 22:40:09.068955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:09.068475 4952 scope.go:122] "RemoveContainer" containerID="67914fcb09125fdd25fd723adbf5ef9f166e604bb58a441ccb6cd0b4ba9cd879" Oct 29 22:40:09.075547 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:09.075512 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:09.129975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:09.129938 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5"] Oct 29 22:40:09.134928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:09.134884 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7fe78e-5f89686c8d-ph9z5"] Oct 29 22:40:09.262998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:09.257527 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:09.637295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:09.637259 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:10.647965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:10.647926 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6c96a8-b18d-4f10-8147-34df971af0c3" path="/var/lib/kubelet/pods/4f6c96a8-b18d-4f10-8147-34df971af0c3/volumes" Oct 29 22:40:11.194932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:11.186508 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:12.838041 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:12.836449 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:13.173628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:13.173538 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:13.368991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:13.368954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:14.200930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:14.200685 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:14.381524 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:14.381478 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:14.491419 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:14.490487 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97"] Oct 29 22:40:14.491419 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:14.490759 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" containerID="cri-o://7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586" gracePeriod=30 Oct 29 22:40:14.553648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:14.553595 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:14.760575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:14.760492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:15.136078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:15.136037 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:15.531058 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:40:15.530963 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:40:16.157957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:16.157920 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:16.686104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:16.686060 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:17.638371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:17.638326 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:17.815897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:17.815849 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:18.111475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:18.111438 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:18.706927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:18.703376 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:19.806130 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.806098 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:40:19.940975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.940932 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istiod-ca-cert\") pod \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " Oct 29 22:40:19.941193 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941047 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-podinfo\") pod \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " Oct 29 22:40:19.941193 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941110 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-token\" (UniqueName: \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-token\") pod \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " Oct 29 22:40:19.941193 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941167 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-certs\") pod \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " Oct 29 22:40:19.941380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941214 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-envoy\") pod \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " Oct 29 22:40:19.941380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941254 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-data\") pod \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " Oct 29 22:40:19.941380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941340 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-credential-socket\") pod \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " Oct 29 22:40:19.941380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941339 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istiod-ca-cert" pod "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" (UID: "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:19.941593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941445 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-socket\") pod \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " Oct 29 22:40:19.941593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941511 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-kube-api-access-jcnln\" (UniqueName: \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-kube-api-access-jcnln\") pod \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\" (UID: \"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b\") " Oct 29 22:40:19.941710 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941672 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-certs" pod "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" (UID: "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:19.941973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.941949 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-data" pod "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" (UID: "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:19.942089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.942021 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-socket" pod "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" (UID: "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:19.942089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.942054 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-credential-socket" pod "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" (UID: "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:19.942704 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.942681 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.942967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.942948 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.943109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.943095 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.943224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.943210 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.943328 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.943310 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:19.944982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.944892 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-token" pod "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" (UID: "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:19.945086 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.945037 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-envoy" pod "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" (UID: "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:19.945368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.945335 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-kube-api-access-jcnln" pod "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" (UID: "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b"). InnerVolumeSpecName "kube-api-access-jcnln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:19.945368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.945354 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-podinfo" pod "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" (UID: "a26f0b9f-c2bd-4394-a3f1-4997686f5a1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:19.968506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.968449 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2"] Oct 29 22:40:19.981570 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:19.981520 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2"] Oct 29 22:40:20.044742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.044654 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.044742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.044701 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.044742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.044716 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.044742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.044730 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcnln\" (UniqueName: \"kubernetes.io/projected/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b-kube-api-access-jcnln\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.153869 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.153829 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:20.199020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.198978 4952 generic.go:356] "Generic (PLEG): container finished" podID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerID="7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586" exitCode=0 Oct 29 22:40:20.199237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.199026 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" event={"ID":"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b","Type":"ContainerDied","Data":"7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586"} Oct 29 22:40:20.199237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.199054 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" Oct 29 22:40:20.199237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.199132 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97" event={"ID":"a26f0b9f-c2bd-4394-a3f1-4997686f5a1b","Type":"ContainerDied","Data":"7460f53e2aa31f4cfc86e4cef56c068d2dccdaefa4e3efddf8516c5c964f8e54"} Oct 29 22:40:20.199237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.199165 4952 scope.go:122] "RemoveContainer" containerID="7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586" Oct 29 22:40:20.213438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.213413 4952 scope.go:122] "RemoveContainer" containerID="7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586" Oct 29 22:40:20.213769 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:40:20.213745 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586\": container with ID starting with 7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586 not found: ID does not exist" containerID="7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586" Oct 29 22:40:20.213865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.213783 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586"} err="failed to get container status \"7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586\": rpc error: code = NotFound desc = could not find container \"7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586\": container with ID starting with 7126a3dd72e2bc777abbf85e301fded3575ffaa14bf1e7d1a4d011b4ac282586 not found: ID does not exist" Oct 29 22:40:20.271022 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.270973 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97"] Oct 29 22:40:20.275281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.275249 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-76c1b7-openshift-default-848cb8b88f-s7p97"] Oct 29 22:40:20.478938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.476423 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:20.650768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:20.650733 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" path="/var/lib/kubelet/pods/a26f0b9f-c2bd-4394-a3f1-4997686f5a1b/volumes" Oct 29 22:40:22.076612 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:22.076577 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:23.719925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:23.718946 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:25.377496 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:25.377454 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:25.548290 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:25.548253 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:25.954741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:25.954699 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:26.126857 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:26.126819 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:26.376976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:26.376934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:26.557284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:26.557252 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:27.024897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:27.024858 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:28.566061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:28.566020 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:29.161697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:29.161659 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:29.328896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:29.328860 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:30.252552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:30.252509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:31.811699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:31.811649 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:33.357025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:33.356980 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:35.037928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.032224 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" containerID="cri-o://5065a022f6898273d2a05456a40366f9aaeb3c3a48e7c5da8d1d9c195614dd24" gracePeriod=45 Oct 29 22:40:35.056575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.047066 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" containerID="cri-o://e55c013e0b8477cccf5128a8eb900bafb619721718f971648bccc84bc595c6fb" gracePeriod=45 Oct 29 22:40:35.056575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.049557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:35.087588 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:40:35.087547 4952 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e982e38_3019_4f3b_ae7d_a7273433a984.slice/crio-e55c013e0b8477cccf5128a8eb900bafb619721718f971648bccc84bc595c6fb.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:40:35.087767 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:40:35.087738 4952 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06fa9242_c768_443f_9a04_5ac39146babf.slice/crio-5065a022f6898273d2a05456a40366f9aaeb3c3a48e7c5da8d1d9c195614dd24.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:40:35.338105 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.338066 4952 generic.go:356] "Generic (PLEG): container finished" podID="06fa9242-c768-443f-9a04-5ac39146babf" containerID="5065a022f6898273d2a05456a40366f9aaeb3c3a48e7c5da8d1d9c195614dd24" exitCode=0 Oct 29 22:40:35.338412 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.338113 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" event={"ID":"06fa9242-c768-443f-9a04-5ac39146babf","Type":"ContainerDied","Data":"5065a022f6898273d2a05456a40366f9aaeb3c3a48e7c5da8d1d9c195614dd24"} Oct 29 22:40:35.340587 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.340560 4952 generic.go:356] "Generic (PLEG): container finished" podID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerID="e55c013e0b8477cccf5128a8eb900bafb619721718f971648bccc84bc595c6fb" exitCode=0 Oct 29 22:40:35.340699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.340614 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" event={"ID":"5e982e38-3019-4f3b-ae7d-a7273433a984","Type":"ContainerDied","Data":"e55c013e0b8477cccf5128a8eb900bafb619721718f971648bccc84bc595c6fb"} Oct 29 22:40:35.379628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.379576 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 22:40:35.411688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.411649 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 22:40:35.471556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.471513 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-model-cache\") pod \"5e982e38-3019-4f3b-ae7d-a7273433a984\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " Oct 29 22:40:35.471829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.471635 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-tmp-dir\") pod \"5e982e38-3019-4f3b-ae7d-a7273433a984\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " Oct 29 22:40:35.472006 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.471978 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-tmp-dir" pod "5e982e38-3019-4f3b-ae7d-a7273433a984" (UID: "5e982e38-3019-4f3b-ae7d-a7273433a984"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.472111 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.471999 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-model-cache" pod "5e982e38-3019-4f3b-ae7d-a7273433a984" (UID: "5e982e38-3019-4f3b-ae7d-a7273433a984"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.472111 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.472064 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-home\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-home\") pod \"5e982e38-3019-4f3b-ae7d-a7273433a984\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " Oct 29 22:40:35.472268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.472248 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-dshm\") pod \"5e982e38-3019-4f3b-ae7d-a7273433a984\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " Oct 29 22:40:35.472403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.472376 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-home" pod "5e982e38-3019-4f3b-ae7d-a7273433a984" (UID: "5e982e38-3019-4f3b-ae7d-a7273433a984"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.472482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.472390 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e982e38-3019-4f3b-ae7d-a7273433a984-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e982e38-3019-4f3b-ae7d-a7273433a984-tls-certs\") pod \"5e982e38-3019-4f3b-ae7d-a7273433a984\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " Oct 29 22:40:35.472865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.472535 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e982e38-3019-4f3b-ae7d-a7273433a984-kube-api-access-7rc5q\" (UniqueName: \"kubernetes.io/projected/5e982e38-3019-4f3b-ae7d-a7273433a984-kube-api-access-7rc5q\") pod \"5e982e38-3019-4f3b-ae7d-a7273433a984\" (UID: \"5e982e38-3019-4f3b-ae7d-a7273433a984\") " Oct 29 22:40:35.473677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.473647 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.473778 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.473708 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.473778 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.473725 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.477297 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.476758 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-dshm" pod "5e982e38-3019-4f3b-ae7d-a7273433a984" (UID: "5e982e38-3019-4f3b-ae7d-a7273433a984"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.479278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.478344 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e982e38-3019-4f3b-ae7d-a7273433a984-kube-api-access-7rc5q" pod "5e982e38-3019-4f3b-ae7d-a7273433a984" (UID: "5e982e38-3019-4f3b-ae7d-a7273433a984"). InnerVolumeSpecName "kube-api-access-7rc5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:35.479278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.479094 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e982e38-3019-4f3b-ae7d-a7273433a984-tls-certs" pod "5e982e38-3019-4f3b-ae7d-a7273433a984" (UID: "5e982e38-3019-4f3b-ae7d-a7273433a984"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:35.574562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.574512 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-dshm\") pod \"06fa9242-c768-443f-9a04-5ac39146babf\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " Oct 29 22:40:35.574562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.574570 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-model-cache\") pod \"06fa9242-c768-443f-9a04-5ac39146babf\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " Oct 29 22:40:35.574928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.574623 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-tmp-dir\") pod \"06fa9242-c768-443f-9a04-5ac39146babf\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " Oct 29 22:40:35.574928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.574649 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-home\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-home\") pod \"06fa9242-c768-443f-9a04-5ac39146babf\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " Oct 29 22:40:35.574928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.574717 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06fa9242-c768-443f-9a04-5ac39146babf-tls-certs\" (UniqueName: \"kubernetes.io/secret/06fa9242-c768-443f-9a04-5ac39146babf-tls-certs\") pod \"06fa9242-c768-443f-9a04-5ac39146babf\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " Oct 29 22:40:35.575117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.574947 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06fa9242-c768-443f-9a04-5ac39146babf-kube-api-access-w9rct\" (UniqueName: \"kubernetes.io/projected/06fa9242-c768-443f-9a04-5ac39146babf-kube-api-access-w9rct\") pod \"06fa9242-c768-443f-9a04-5ac39146babf\" (UID: \"06fa9242-c768-443f-9a04-5ac39146babf\") " Oct 29 22:40:35.575117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.574944 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-model-cache" pod "06fa9242-c768-443f-9a04-5ac39146babf" (UID: "06fa9242-c768-443f-9a04-5ac39146babf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.575117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.574968 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-tmp-dir" pod "06fa9242-c768-443f-9a04-5ac39146babf" (UID: "06fa9242-c768-443f-9a04-5ac39146babf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.575117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.575036 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-home" pod "06fa9242-c768-443f-9a04-5ac39146babf" (UID: "06fa9242-c768-443f-9a04-5ac39146babf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.575532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.575511 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e982e38-3019-4f3b-ae7d-a7273433a984-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.575611 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.575534 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rc5q\" (UniqueName: \"kubernetes.io/projected/5e982e38-3019-4f3b-ae7d-a7273433a984-kube-api-access-7rc5q\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.575611 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.575549 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.575611 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.575563 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.575611 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.575574 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.575611 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.575587 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e982e38-3019-4f3b-ae7d-a7273433a984-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.577166 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.577139 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06fa9242-c768-443f-9a04-5ac39146babf-tls-certs" pod "06fa9242-c768-443f-9a04-5ac39146babf" (UID: "06fa9242-c768-443f-9a04-5ac39146babf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:40:35.577317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.577247 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-dshm" pod "06fa9242-c768-443f-9a04-5ac39146babf" (UID: "06fa9242-c768-443f-9a04-5ac39146babf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:35.577317 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.577259 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fa9242-c768-443f-9a04-5ac39146babf-kube-api-access-w9rct" pod "06fa9242-c768-443f-9a04-5ac39146babf" (UID: "06fa9242-c768-443f-9a04-5ac39146babf"). InnerVolumeSpecName "kube-api-access-w9rct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:35.677729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.677645 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rct\" (UniqueName: \"kubernetes.io/projected/06fa9242-c768-443f-9a04-5ac39146babf-kube-api-access-w9rct\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.677729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.677679 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06fa9242-c768-443f-9a04-5ac39146babf-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:35.677729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:35.677691 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06fa9242-c768-443f-9a04-5ac39146babf-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:40:36.331830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.331785 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 29 22:40:36.350250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.350165 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" Oct 29 22:40:36.350250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.350196 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2" event={"ID":"06fa9242-c768-443f-9a04-5ac39146babf","Type":"ContainerDied","Data":"4346243403ac410a806407fb8133b3e1d6d9a6aaa4c664727ff0c213e85f4729"} Oct 29 22:40:36.350499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.350258 4952 scope.go:122] "RemoveContainer" containerID="5065a022f6898273d2a05456a40366f9aaeb3c3a48e7c5da8d1d9c195614dd24" Oct 29 22:40:36.352089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.352065 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" Oct 29 22:40:36.352233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.352161 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2" event={"ID":"5e982e38-3019-4f3b-ae7d-a7273433a984","Type":"ContainerDied","Data":"f732a5535aab950b3b229520db2831ad91feda0d0868bfb56f17363846bc8ffe"} Oct 29 22:40:36.364135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.364005 4952 scope.go:122] "RemoveContainer" containerID="e55c013e0b8477cccf5128a8eb900bafb619721718f971648bccc84bc595c6fb" Oct 29 22:40:36.415669 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.415629 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2"] Oct 29 22:40:36.431167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.431127 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-2-w2-92d984-kserve-5897df59d8-dkdl2"] Oct 29 22:40:36.463017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.462983 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2"] Oct 29 22:40:36.469953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.469921 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-92d984/distinct-w2-92d984-kserve-7d48cb8547-6zcs2"] Oct 29 22:40:36.579495 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.579075 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:36.648113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.648027 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06fa9242-c768-443f-9a04-5ac39146babf" path="/var/lib/kubelet/pods/06fa9242-c768-443f-9a04-5ac39146babf/volumes" Oct 29 22:40:36.648581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:36.648564 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" path="/var/lib/kubelet/pods/5e982e38-3019-4f3b-ae7d-a7273433a984/volumes" Oct 29 22:40:38.296930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:38.288818 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:39.646035 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:39.645995 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:40.345099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:40.345063 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:40.531505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:40.531467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:40.780523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:40.780413 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:41.116735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:41.116697 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:42.418176 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:42.417481 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:43.799669 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:43.799624 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:44.475241 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:44.475200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:44.650652 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:44.650604 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:44.820047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:44.819884 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:45.120708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:45.120654 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:45.532699 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:40:45.532606 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:40:46.443929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:46.442036 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:48.210591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:48.210536 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:48.385327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:48.385292 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:48.637584 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:48.637552 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:48.815985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:48.815945 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:49.941608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:49.941572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:51.733748 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:51.733647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:52.209991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:52.209893 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:52.378350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:52.378306 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:52.567689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:52.567582 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:53.653167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:53.653123 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:55.441954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:55.441892 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:57.331960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:57.328991 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:57.943423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:57.942608 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:40:57.957627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:57.957405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:58.148041 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:58.142668 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:59.167619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:40:59.167580 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:00.249845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:00.249704 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:00.454934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:00.451488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:00.725941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:00.723284 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:01.594931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:01.591682 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:02.028300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:02.028206 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:03.672928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:03.669303 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:03.890239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:03.889978 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:04.084490 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:04.084447 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:05.652763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:05.652707 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:07.404090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:07.403991 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:07.572327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:07.572237 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:09.108473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:09.108357 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:09.600100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:09.600055 4952 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:41:09.602246 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:09.600140 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:41:09.902564 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:09.902474 4952 generic.go:356] "Generic (PLEG): container finished" podID="6862987c-7402-4134-a7a7-4a1663aed0de" containerID="62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39" exitCode=0 Oct 29 22:41:09.902754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:09.902553 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" event={"ID":"6862987c-7402-4134-a7a7-4a1663aed0de","Type":"ContainerDied","Data":"62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39"} Oct 29 22:41:09.903075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:09.903056 4952 scope.go:122] "RemoveContainer" containerID="62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39" Oct 29 22:41:09.905132 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:09.905104 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:41:09.905218 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:09.905181 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:41:09.905218 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:09.905215 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:41:10.713926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:10.711348 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:10.912608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:10.912574 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" event={"ID":"6862987c-7402-4134-a7a7-4a1663aed0de","Type":"ContainerStarted","Data":"7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b"} Oct 29 22:41:11.810816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:11.810777 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:41:11.811293 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:11.811014 4952 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:41:11.811293 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:11.811077 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 29 22:41:12.433565 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:12.433529 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:12.810672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:12.810580 4952 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:41:12.810672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:12.810644 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 29 22:41:13.307474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:13.307438 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:13.790634 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:13.790592 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm"] Oct 29 22:41:13.790957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:13.790922 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" podUID="9589038c-da0d-4e10-8a0c-05d93e455357" containerName="maas-api" containerID="cri-o://6ef1ba1ef02280c6a2133c686e558a43ff8c148cab372a78867293aa5c2f8a08" gracePeriod=30 Oct 29 22:41:13.869098 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:13.855749 4952 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:41:13.869098 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:13.855821 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 29 22:41:13.911511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:13.911281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:13.986446 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:13.986170 4952 generic.go:356] "Generic (PLEG): container finished" podID="9589038c-da0d-4e10-8a0c-05d93e455357" containerID="6ef1ba1ef02280c6a2133c686e558a43ff8c148cab372a78867293aa5c2f8a08" exitCode=0 Oct 29 22:41:13.986446 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:13.986375 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" event={"ID":"9589038c-da0d-4e10-8a0c-05d93e455357","Type":"ContainerDied","Data":"6ef1ba1ef02280c6a2133c686e558a43ff8c148cab372a78867293aa5c2f8a08"} Oct 29 22:41:14.689800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.689765 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:41:14.755633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.754929 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:14.814936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.814055 4952 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:41:14.814936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.814176 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 29 22:41:14.885239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.883860 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9589038c-da0d-4e10-8a0c-05d93e455357-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9589038c-da0d-4e10-8a0c-05d93e455357-openshift-service-ca\") pod \"9589038c-da0d-4e10-8a0c-05d93e455357\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " Oct 29 22:41:14.885239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.883976 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-metrics-certs\") pod \"9589038c-da0d-4e10-8a0c-05d93e455357\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " Oct 29 22:41:14.885239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.884078 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-maas-api-tls\") pod \"9589038c-da0d-4e10-8a0c-05d93e455357\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " Oct 29 22:41:14.885239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.884165 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9589038c-da0d-4e10-8a0c-05d93e455357-kube-api-access-4tbkw\" (UniqueName: \"kubernetes.io/projected/9589038c-da0d-4e10-8a0c-05d93e455357-kube-api-access-4tbkw\") pod \"9589038c-da0d-4e10-8a0c-05d93e455357\" (UID: \"9589038c-da0d-4e10-8a0c-05d93e455357\") " Oct 29 22:41:14.905114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.905048 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-maas-api-tls" pod "9589038c-da0d-4e10-8a0c-05d93e455357" (UID: "9589038c-da0d-4e10-8a0c-05d93e455357"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:14.910847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.910802 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9589038c-da0d-4e10-8a0c-05d93e455357-kube-api-access-4tbkw" pod "9589038c-da0d-4e10-8a0c-05d93e455357" (UID: "9589038c-da0d-4e10-8a0c-05d93e455357"). InnerVolumeSpecName "kube-api-access-4tbkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:14.913648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.913610 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9589038c-da0d-4e10-8a0c-05d93e455357-openshift-service-ca" pod "9589038c-da0d-4e10-8a0c-05d93e455357" (UID: "9589038c-da0d-4e10-8a0c-05d93e455357"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:14.916049 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.916005 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-metrics-certs" pod "9589038c-da0d-4e10-8a0c-05d93e455357" (UID: "9589038c-da0d-4e10-8a0c-05d93e455357"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:14.948925 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.947051 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:14.986044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.986008 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:14.986044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.986036 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9589038c-da0d-4e10-8a0c-05d93e455357-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:14.986044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.986046 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tbkw\" (UniqueName: \"kubernetes.io/projected/9589038c-da0d-4e10-8a0c-05d93e455357-kube-api-access-4tbkw\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:14.987147 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.986056 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9589038c-da0d-4e10-8a0c-05d93e455357-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:14.998878 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.998852 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" Oct 29 22:41:14.999043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.998929 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm" event={"ID":"9589038c-da0d-4e10-8a0c-05d93e455357","Type":"ContainerDied","Data":"0a92c78bb214e8b4fe0f0e06ec3da73e6b64dbf237bc8863b12854e6b7945e7e"} Oct 29 22:41:14.999043 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:14.999014 4952 scope.go:122] "RemoveContainer" containerID="6ef1ba1ef02280c6a2133c686e558a43ff8c148cab372a78867293aa5c2f8a08" Oct 29 22:41:15.112786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:15.112708 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm"] Oct 29 22:41:15.117935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:15.115216 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c1015-5f794f6687-6qbsm"] Oct 29 22:41:15.119931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:15.119322 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:15.464176 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:15.463177 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:15.560382 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:41:15.560335 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:41:15.810656 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:15.810575 4952 patch_prober.go:29] interesting pod/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 29 22:41:15.810656 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:15.810632 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 29 22:41:16.606366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.606328 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq"] Oct 29 22:41:16.607063 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607033 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerName="istio-proxy" Oct 29 22:41:16.607063 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607056 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerName="istio-proxy" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607080 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4f6c96a8-b18d-4f10-8147-34df971af0c3" containerName="maas-api" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607088 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6c96a8-b18d-4f10-8147-34df971af0c3" containerName="maas-api" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607105 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607113 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607127 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9589038c-da0d-4e10-8a0c-05d93e455357" containerName="maas-api" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607134 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9589038c-da0d-4e10-8a0c-05d93e455357" containerName="maas-api" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607155 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607162 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607201 4952 cpu_manager.go:414] "containerMap: removing container" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607208 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607225 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607232 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607254 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" Oct 29 22:41:16.607310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607262 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" Oct 29 22:41:16.608249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607430 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6add6ee-6c29-4d7e-94f1-ddb3698c1244" containerName="istio-proxy" Oct 29 22:41:16.608249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607452 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6c96a8-b18d-4f10-8147-34df971af0c3" containerName="maas-api" Oct 29 22:41:16.608249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607462 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e982e38-3019-4f3b-ae7d-a7273433a984" containerName="main" Oct 29 22:41:16.608249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607474 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9589038c-da0d-4e10-8a0c-05d93e455357" containerName="maas-api" Oct 29 22:41:16.608249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607486 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26f0b9f-c2bd-4394-a3f1-4997686f5a1b" containerName="istio-proxy" Oct 29 22:41:16.608249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607499 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="06fa9242-c768-443f-9a04-5ac39146babf" containerName="main" Oct 29 22:41:16.608249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607508 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" Oct 29 22:41:16.608249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.607521 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f904684-fe8c-4e96-befb-229dbf30522f" containerName="istio-proxy" Oct 29 22:41:16.612693 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.612663 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.661980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.657548 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d09411-openshift-default-dockercfg-rwwzk\"" Oct 29 22:41:16.687188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.687150 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9589038c-da0d-4e10-8a0c-05d93e455357" path="/var/lib/kubelet/pods/9589038c-da0d-4e10-8a0c-05d93e455357/volumes" Oct 29 22:41:16.687622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.687601 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq"] Oct 29 22:41:16.712453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.712415 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ccaa807-dd67-431a-8b48-07cf649eefed-istiod-ca-cert\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.712636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.712499 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-envoy\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.712636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.712538 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-data\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.712728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.712639 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrz4\" (UniqueName: \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-kube-api-access-zdrz4\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.712789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.712775 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-certs\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.712829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.712819 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-podinfo\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.712955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.712939 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-token\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.713005 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.712979 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.713096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.713073 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-credential-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.813971 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.813935 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-envoy\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.814150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.813979 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-data\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.814150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.814009 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrz4\" (UniqueName: \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-kube-api-access-zdrz4\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.814150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.814134 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-certs\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.814315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.814188 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-podinfo\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.814315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.814256 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-token\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.814315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.814307 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.814484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.814366 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-credential-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.814484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.814426 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-data\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.814590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.814486 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ccaa807-dd67-431a-8b48-07cf649eefed-istiod-ca-cert\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.814666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.814642 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-certs\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.815016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.814987 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.815281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.815083 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-credential-socket\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.815455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.815359 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ccaa807-dd67-431a-8b48-07cf649eefed-istiod-ca-cert\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.816863 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.816843 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:41:16.817173 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.817150 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-envoy\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.817276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.817256 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-podinfo\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.817359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.817342 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:41:16.818255 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.818230 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:41:16.839972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.839937 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-token\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.840835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.840815 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrz4\" (UniqueName: \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-kube-api-access-zdrz4\") pod \"e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:16.929865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:16.929125 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:17.008074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:17.008035 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:17.275788 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:17.275754 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq"] Oct 29 22:41:17.276653 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:41:17.276629 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ccaa807_dd67_431a_8b48_07cf649eefed.slice/crio-b595be7358b2c57679709f55eb1b68e293fce178342cb93d72b149dc7a062824 WatchSource:0}: Error finding container b595be7358b2c57679709f55eb1b68e293fce178342cb93d72b149dc7a062824: Status 404 returned error can't find the container with id b595be7358b2c57679709f55eb1b68e293fce178342cb93d72b149dc7a062824 Oct 29 22:41:17.279535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:17.279499 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:41:17.279651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:17.279584 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:41:17.279651 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:17.279630 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 29 22:41:18.030196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:18.030152 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" event={"ID":"1ccaa807-dd67-431a-8b48-07cf649eefed","Type":"ContainerStarted","Data":"836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40"} Oct 29 22:41:18.030721 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:18.030217 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" event={"ID":"1ccaa807-dd67-431a-8b48-07cf649eefed","Type":"ContainerStarted","Data":"b595be7358b2c57679709f55eb1b68e293fce178342cb93d72b149dc7a062824"} Oct 29 22:41:18.092715 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:18.092651 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" podStartSLOduration=2.0926357700000002 podStartE2EDuration="2.09263577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:18.091042829 +0000 UTC m=+3543.565631250" watchObservedRunningTime="2026-10-29 22:41:18.09263577 +0000 UTC m=+3543.567224188" Oct 29 22:41:18.639763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:18.639722 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:18.937997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:18.931974 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:18.948928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:18.940029 4952 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 29 22:41:18.948928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:18.940101 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 29 22:41:19.943930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:19.932501 4952 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 29 22:41:19.943930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:19.932584 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 29 22:41:20.355961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:20.347229 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:20.941927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:20.933308 4952 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 29 22:41:20.941927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:20.933377 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 29 22:41:21.936931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:21.936081 4952 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 29 22:41:21.936931 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:21.936205 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 29 22:41:22.153930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:22.141666 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:22.357932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:22.355667 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:22.730010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:22.729258 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:23.007029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:23.000726 4952 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 29 22:41:23.007612 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:23.007547 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 29 22:41:23.942929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:23.942295 4952 patch_prober.go:29] interesting pod/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 29 22:41:23.942929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:23.942358 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 29 22:41:24.454930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:24.454518 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:24.979321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:24.979284 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:24.979888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:24.979859 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:24.981104 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:24.981085 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:41:26.108734 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:26.108502 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:27.822851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:27.822806 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:27.993151 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:27.993109 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:29.451932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:29.451506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:31.010640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:31.010435 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:31.953511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:31.953464 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:32.992710 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:32.992667 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:33.757735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:33.757695 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:33.941569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:33.941528 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:34.816204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:34.816159 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:36.354094 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:36.353840 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:37.710119 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:37.710079 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:38.440818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:38.440769 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:38.628365 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:38.628313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:38.803233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:38.803137 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:39.074590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:39.074546 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:40.046517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.044729 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7"] Oct 29 22:41:40.047370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.047333 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" podUID="1deeeb21-199d-4f40-81c6-192ce3dbb82a" containerName="maas-api" containerID="cri-o://4faf8451d4431ab1fc7719c5171dd069e852389bdea8872106aae6d5128feb92" gracePeriod=30 Oct 29 22:41:40.216869 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.216817 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:40.740920 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.740828 4952 generic.go:356] "Generic (PLEG): container finished" podID="1deeeb21-199d-4f40-81c6-192ce3dbb82a" containerID="4faf8451d4431ab1fc7719c5171dd069e852389bdea8872106aae6d5128feb92" exitCode=0 Oct 29 22:41:40.750750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.750709 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" event={"ID":"1deeeb21-199d-4f40-81c6-192ce3dbb82a","Type":"ContainerDied","Data":"4faf8451d4431ab1fc7719c5171dd069e852389bdea8872106aae6d5128feb92"} Oct 29 22:41:40.767473 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.767445 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:41:40.869765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.869723 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-maas-api-tls\") pod \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " Oct 29 22:41:40.870120 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.870092 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-metrics-certs\") pod \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " Oct 29 22:41:40.870276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.870261 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1deeeb21-199d-4f40-81c6-192ce3dbb82a-kube-api-access-4625q\" (UniqueName: \"kubernetes.io/projected/1deeeb21-199d-4f40-81c6-192ce3dbb82a-kube-api-access-4625q\") pod \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " Oct 29 22:41:40.870493 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.870477 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1deeeb21-199d-4f40-81c6-192ce3dbb82a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1deeeb21-199d-4f40-81c6-192ce3dbb82a-openshift-service-ca\") pod \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\" (UID: \"1deeeb21-199d-4f40-81c6-192ce3dbb82a\") " Oct 29 22:41:40.873095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.873059 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1deeeb21-199d-4f40-81c6-192ce3dbb82a-openshift-service-ca" pod "1deeeb21-199d-4f40-81c6-192ce3dbb82a" (UID: "1deeeb21-199d-4f40-81c6-192ce3dbb82a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:40.875743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.875716 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-metrics-certs" pod "1deeeb21-199d-4f40-81c6-192ce3dbb82a" (UID: "1deeeb21-199d-4f40-81c6-192ce3dbb82a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:40.876510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.876491 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-maas-api-tls" pod "1deeeb21-199d-4f40-81c6-192ce3dbb82a" (UID: "1deeeb21-199d-4f40-81c6-192ce3dbb82a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:40.876602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.876574 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1deeeb21-199d-4f40-81c6-192ce3dbb82a-kube-api-access-4625q" pod "1deeeb21-199d-4f40-81c6-192ce3dbb82a" (UID: "1deeeb21-199d-4f40-81c6-192ce3dbb82a"). InnerVolumeSpecName "kube-api-access-4625q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:40.973448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.973349 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1deeeb21-199d-4f40-81c6-192ce3dbb82a-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:40.973448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.973391 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:40.973448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.973403 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1deeeb21-199d-4f40-81c6-192ce3dbb82a-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:40.973448 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:40.973415 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4625q\" (UniqueName: \"kubernetes.io/projected/1deeeb21-199d-4f40-81c6-192ce3dbb82a-kube-api-access-4625q\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:41.111625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:41.111583 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:41.277639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:41.277537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:41.712047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:41.712003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:41.752450 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:41.752412 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" Oct 29 22:41:41.752648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:41.752421 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" event={"ID":"1deeeb21-199d-4f40-81c6-192ce3dbb82a","Type":"ContainerDied","Data":"490b0ac572d2844e814b3a0ee1c0c3587a33cec99789f5244f6fb89d07758f6f"} Oct 29 22:41:41.752648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:41.752575 4952 scope.go:122] "RemoveContainer" containerID="4faf8451d4431ab1fc7719c5171dd069e852389bdea8872106aae6d5128feb92" Oct 29 22:41:41.814267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:41.814228 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7"] Oct 29 22:41:41.827131 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:41.827090 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7"] Oct 29 22:41:42.545840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:42.545802 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th"] Oct 29 22:41:42.546365 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:42.546072 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" podUID="9f169249-a54d-45cc-a71e-52a5bd54d81d" containerName="maas-api" containerID="cri-o://2ef6aba0a567a7d7cdc2df524ffe2f0fea1917564d06827b874b3191c09a43f6" gracePeriod=30 Oct 29 22:41:42.649720 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:42.649685 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1deeeb21-199d-4f40-81c6-192ce3dbb82a" path="/var/lib/kubelet/pods/1deeeb21-199d-4f40-81c6-192ce3dbb82a/volumes" Oct 29 22:41:42.661605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:42.661567 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:42.977743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:42.977448 4952 generic.go:356] "Generic (PLEG): container finished" podID="9f169249-a54d-45cc-a71e-52a5bd54d81d" containerID="2ef6aba0a567a7d7cdc2df524ffe2f0fea1917564d06827b874b3191c09a43f6" exitCode=0 Oct 29 22:41:42.977743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:42.977669 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" event={"ID":"9f169249-a54d-45cc-a71e-52a5bd54d81d","Type":"ContainerDied","Data":"2ef6aba0a567a7d7cdc2df524ffe2f0fea1917564d06827b874b3191c09a43f6"} Oct 29 22:41:43.209586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.208299 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:41:43.333170 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.333105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:43.359482 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.359439 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-maas-api-tls\") pod \"9f169249-a54d-45cc-a71e-52a5bd54d81d\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " Oct 29 22:41:43.359689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.359489 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f169249-a54d-45cc-a71e-52a5bd54d81d-kube-api-access-xvft2\" (UniqueName: \"kubernetes.io/projected/9f169249-a54d-45cc-a71e-52a5bd54d81d-kube-api-access-xvft2\") pod \"9f169249-a54d-45cc-a71e-52a5bd54d81d\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " Oct 29 22:41:43.359689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.359560 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f169249-a54d-45cc-a71e-52a5bd54d81d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f169249-a54d-45cc-a71e-52a5bd54d81d-openshift-service-ca\") pod \"9f169249-a54d-45cc-a71e-52a5bd54d81d\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " Oct 29 22:41:43.359689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.359624 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-metrics-certs\") pod \"9f169249-a54d-45cc-a71e-52a5bd54d81d\" (UID: \"9f169249-a54d-45cc-a71e-52a5bd54d81d\") " Oct 29 22:41:43.360359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.360188 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f169249-a54d-45cc-a71e-52a5bd54d81d-openshift-service-ca" pod "9f169249-a54d-45cc-a71e-52a5bd54d81d" (UID: "9f169249-a54d-45cc-a71e-52a5bd54d81d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:43.362326 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.362297 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f169249-a54d-45cc-a71e-52a5bd54d81d-kube-api-access-xvft2" pod "9f169249-a54d-45cc-a71e-52a5bd54d81d" (UID: "9f169249-a54d-45cc-a71e-52a5bd54d81d"). InnerVolumeSpecName "kube-api-access-xvft2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:43.362465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.362382 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-metrics-certs" pod "9f169249-a54d-45cc-a71e-52a5bd54d81d" (UID: "9f169249-a54d-45cc-a71e-52a5bd54d81d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:43.362465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.362399 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-maas-api-tls" pod "9f169249-a54d-45cc-a71e-52a5bd54d81d" (UID: "9f169249-a54d-45cc-a71e-52a5bd54d81d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:43.461742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.461702 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f169249-a54d-45cc-a71e-52a5bd54d81d-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:43.461742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.461737 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:43.461742 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.461746 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f169249-a54d-45cc-a71e-52a5bd54d81d-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:43.462110 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.461755 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvft2\" (UniqueName: \"kubernetes.io/projected/9f169249-a54d-45cc-a71e-52a5bd54d81d-kube-api-access-xvft2\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:43.517930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.516926 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-798a8f-7b486b9b79-cqhv7" podUID="1deeeb21-199d-4f40-81c6-192ce3dbb82a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:41:43.528928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.525565 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:43.691755 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.691653 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:43.959165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.959068 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:43.986779 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.986751 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" Oct 29 22:41:43.987030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.986741 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th" event={"ID":"9f169249-a54d-45cc-a71e-52a5bd54d81d","Type":"ContainerDied","Data":"4070921eeeaa6457753e185dbee727bfca8abdb49c65f3984ae6a535f86c80c0"} Oct 29 22:41:43.987030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:43.986896 4952 scope.go:122] "RemoveContainer" containerID="2ef6aba0a567a7d7cdc2df524ffe2f0fea1917564d06827b874b3191c09a43f6" Oct 29 22:41:44.046594 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:44.046548 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th"] Oct 29 22:41:44.054357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:44.054318 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-675bf4-7f4688fcfc-gw5th"] Oct 29 22:41:44.649159 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:44.649119 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f169249-a54d-45cc-a71e-52a5bd54d81d" path="/var/lib/kubelet/pods/9f169249-a54d-45cc-a71e-52a5bd54d81d/volumes" Oct 29 22:41:45.315310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:45.315266 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:45.540633 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:41:45.540589 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:41:46.145771 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:46.145731 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:46.312257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:46.312218 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:46.443253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:46.443166 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6"] Oct 29 22:41:46.443975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:46.443937 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" containerID="cri-o://5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5" gracePeriod=30 Oct 29 22:41:46.621926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:46.597964 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:47.050489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:47.050422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:47.849237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:47.849198 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:41:47.849730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:47.849275 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:41:48.636702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:48.636657 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:48.883233 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:48.883181 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:49.063637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:49.063529 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:49.231136 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:49.231090 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:49.746440 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:49.746393 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:50.489089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:50.486126 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:51.232343 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:51.232299 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:51.402011 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:51.401973 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:51.583896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:51.582347 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:51.962973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:51.961248 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 22:41:52.077933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.077225 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e290a6d-f30b-4005-b567-94091995ec44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e290a6d-f30b-4005-b567-94091995ec44-istiod-ca-cert\") pod \"9e290a6d-f30b-4005-b567-94091995ec44\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " Oct 29 22:41:52.077933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.077284 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-certs\") pod \"9e290a6d-f30b-4005-b567-94091995ec44\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " Oct 29 22:41:52.077933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.077330 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-envoy\") pod \"9e290a6d-f30b-4005-b567-94091995ec44\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " Oct 29 22:41:52.077933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.077407 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e290a6d-f30b-4005-b567-94091995ec44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e290a6d-f30b-4005-b567-94091995ec44-istio-podinfo\") pod \"9e290a6d-f30b-4005-b567-94091995ec44\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " Oct 29 22:41:52.077933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.077532 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-credential-socket\") pod \"9e290a6d-f30b-4005-b567-94091995ec44\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " Oct 29 22:41:52.077933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.077601 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-data\") pod \"9e290a6d-f30b-4005-b567-94091995ec44\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " Oct 29 22:41:52.077933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.077621 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e290a6d-f30b-4005-b567-94091995ec44-istiod-ca-cert" pod "9e290a6d-f30b-4005-b567-94091995ec44" (UID: "9e290a6d-f30b-4005-b567-94091995ec44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:52.077933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.077645 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-socket\") pod \"9e290a6d-f30b-4005-b567-94091995ec44\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " Oct 29 22:41:52.077933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.077697 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-istio-token\" (UniqueName: \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-istio-token\") pod \"9e290a6d-f30b-4005-b567-94091995ec44\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " Oct 29 22:41:52.077933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.077752 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-kube-api-access-fjgc6\" (UniqueName: \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-kube-api-access-fjgc6\") pod \"9e290a6d-f30b-4005-b567-94091995ec44\" (UID: \"9e290a6d-f30b-4005-b567-94091995ec44\") " Oct 29 22:41:52.079350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.078195 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-data" pod "9e290a6d-f30b-4005-b567-94091995ec44" (UID: "9e290a6d-f30b-4005-b567-94091995ec44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.079350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.078265 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-certs" pod "9e290a6d-f30b-4005-b567-94091995ec44" (UID: "9e290a6d-f30b-4005-b567-94091995ec44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.079350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.078373 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-credential-socket" pod "9e290a6d-f30b-4005-b567-94091995ec44" (UID: "9e290a6d-f30b-4005-b567-94091995ec44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.079350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.078429 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-socket" pod "9e290a6d-f30b-4005-b567-94091995ec44" (UID: "9e290a6d-f30b-4005-b567-94091995ec44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.079350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.078895 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.079350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.078941 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.079350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.078955 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.079350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.078970 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e290a6d-f30b-4005-b567-94091995ec44-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.079350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.078983 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.080560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.080496 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e290a6d-f30b-4005-b567-94091995ec44-istio-podinfo" pod "9e290a6d-f30b-4005-b567-94091995ec44" (UID: "9e290a6d-f30b-4005-b567-94091995ec44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:41:52.080887 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.080856 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-istio-token" pod "9e290a6d-f30b-4005-b567-94091995ec44" (UID: "9e290a6d-f30b-4005-b567-94091995ec44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:52.080997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.080921 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-kube-api-access-fjgc6" pod "9e290a6d-f30b-4005-b567-94091995ec44" (UID: "9e290a6d-f30b-4005-b567-94091995ec44"). InnerVolumeSpecName "kube-api-access-fjgc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:52.082333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.082297 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-envoy" pod "9e290a6d-f30b-4005-b567-94091995ec44" (UID: "9e290a6d-f30b-4005-b567-94091995ec44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:41:52.098222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.098190 4952 generic.go:356] "Generic (PLEG): container finished" podID="9e290a6d-f30b-4005-b567-94091995ec44" containerID="5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5" exitCode=0 Oct 29 22:41:52.098379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.098251 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" Oct 29 22:41:52.098379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.098278 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" event={"ID":"9e290a6d-f30b-4005-b567-94091995ec44","Type":"ContainerDied","Data":"5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5"} Oct 29 22:41:52.098379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.098340 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6" event={"ID":"9e290a6d-f30b-4005-b567-94091995ec44","Type":"ContainerDied","Data":"f2378b23d30edd6347a368df38547a75aacef4dbd4bdcd24822eaccacdb2618a"} Oct 29 22:41:52.098379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.098365 4952 scope.go:122] "RemoveContainer" containerID="5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5" Oct 29 22:41:52.103561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.103526 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:52.119249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.119220 4952 scope.go:122] "RemoveContainer" containerID="5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5" Oct 29 22:41:52.119603 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:41:52.119579 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5\": container with ID starting with 5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5 not found: ID does not exist" containerID="5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5" Oct 29 22:41:52.119667 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.119614 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5"} err="failed to get container status \"5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5\": rpc error: code = NotFound desc = could not find container \"5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5\": container with ID starting with 5c0fdb8753b71e71d6993b8477d411031b151a606065e5a7134fd6468f2da8e5 not found: ID does not exist" Oct 29 22:41:52.175944 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.175456 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6"] Oct 29 22:41:52.180029 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.179991 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-92d984-openshift-default-75b6859fb9-rsnb6"] Oct 29 22:41:52.180316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.180291 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.180363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.180320 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjgc6\" (UniqueName: \"kubernetes.io/projected/9e290a6d-f30b-4005-b567-94091995ec44-kube-api-access-fjgc6\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.180363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.180332 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e290a6d-f30b-4005-b567-94091995ec44-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.180363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.180341 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e290a6d-f30b-4005-b567-94091995ec44-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:41:52.391547 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.391442 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:52.648546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:52.648458 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e290a6d-f30b-4005-b567-94091995ec44" path="/var/lib/kubelet/pods/9e290a6d-f30b-4005-b567-94091995ec44/volumes" Oct 29 22:41:54.245281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:54.240189 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:54.676357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:54.676319 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66"] Oct 29 22:41:54.689527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:54.689493 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv"] Oct 29 22:41:54.893785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:54.893743 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:56.029592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:56.029493 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:57.014492 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:57.014450 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65"] Oct 29 22:41:57.225991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:57.225953 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:57.783927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:57.783758 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:59.443929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:41:59.440931 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:00.591301 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:00.591265 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:00.813636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:00.813600 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:01.104941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:01.099506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:02.152929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:02.152142 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:42:02.911169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:02.911132 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:03.167306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:03.167197 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:03.395831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:03.395786 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:04.737222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:04.737177 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:06.337211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:06.337172 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:08.051084 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:08.051035 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:09.700001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:09.699960 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" containerID="cri-o://64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3" gracePeriod=45 Oct 29 22:42:09.711149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:09.711111 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" containerID="cri-o://28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490" gracePeriod=45 Oct 29 22:42:09.826268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:09.826140 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:10.053585 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.053548 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:42:10.061537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.061364 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 22:42:10.106538 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.106499 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52740807-cfc4-4144-b4dc-14ef28aff6d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/52740807-cfc4-4144-b4dc-14ef28aff6d6-tls-certs\") pod \"52740807-cfc4-4144-b4dc-14ef28aff6d6\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " Oct 29 22:42:10.106743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.106572 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-home\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-home\") pod \"52740807-cfc4-4144-b4dc-14ef28aff6d6\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " Oct 29 22:42:10.106743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.106607 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-tmp-dir\") pod \"52740807-cfc4-4144-b4dc-14ef28aff6d6\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " Oct 29 22:42:10.106743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.106675 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tmp-dir\") pod \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " Oct 29 22:42:10.106743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.106709 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de8d8749-f5c9-4bbc-870a-bd919a44dec7-kube-api-access-gl4jf\" (UniqueName: \"kubernetes.io/projected/de8d8749-f5c9-4bbc-870a-bd919a44dec7-kube-api-access-gl4jf\") pod \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " Oct 29 22:42:10.106896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.106788 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-dshm\") pod \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " Oct 29 22:42:10.106896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.106829 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tls-certs\" (UniqueName: \"kubernetes.io/secret/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tls-certs\") pod \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " Oct 29 22:42:10.106896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.106879 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-home\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-home\") pod \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " Oct 29 22:42:10.107055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.106938 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-dshm\") pod \"52740807-cfc4-4144-b4dc-14ef28aff6d6\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " Oct 29 22:42:10.107055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.107025 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52740807-cfc4-4144-b4dc-14ef28aff6d6-kube-api-access-x8jd8\" (UniqueName: \"kubernetes.io/projected/52740807-cfc4-4144-b4dc-14ef28aff6d6-kube-api-access-x8jd8\") pod \"52740807-cfc4-4144-b4dc-14ef28aff6d6\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " Oct 29 22:42:10.107137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.107073 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-model-cache\") pod \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\" (UID: \"de8d8749-f5c9-4bbc-870a-bd919a44dec7\") " Oct 29 22:42:10.107137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.107124 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-model-cache\") pod \"52740807-cfc4-4144-b4dc-14ef28aff6d6\" (UID: \"52740807-cfc4-4144-b4dc-14ef28aff6d6\") " Oct 29 22:42:10.107826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.107375 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-home" pod "52740807-cfc4-4144-b4dc-14ef28aff6d6" (UID: "52740807-cfc4-4144-b4dc-14ef28aff6d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.107826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.107531 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-tmp-dir" pod "52740807-cfc4-4144-b4dc-14ef28aff6d6" (UID: "52740807-cfc4-4144-b4dc-14ef28aff6d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.107826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.107595 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-home" pod "de8d8749-f5c9-4bbc-870a-bd919a44dec7" (UID: "de8d8749-f5c9-4bbc-870a-bd919a44dec7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.108189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.107875 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.108189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.107897 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.108189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.107927 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.108189 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.108141 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-model-cache" pod "de8d8749-f5c9-4bbc-870a-bd919a44dec7" (UID: "de8d8749-f5c9-4bbc-870a-bd919a44dec7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.108390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.108199 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tmp-dir" pod "de8d8749-f5c9-4bbc-870a-bd919a44dec7" (UID: "de8d8749-f5c9-4bbc-870a-bd919a44dec7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.108390 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.108369 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-model-cache" pod "52740807-cfc4-4144-b4dc-14ef28aff6d6" (UID: "52740807-cfc4-4144-b4dc-14ef28aff6d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.113474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.113382 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52740807-cfc4-4144-b4dc-14ef28aff6d6-tls-certs" pod "52740807-cfc4-4144-b4dc-14ef28aff6d6" (UID: "52740807-cfc4-4144-b4dc-14ef28aff6d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:10.113474 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.113424 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tls-certs" pod "de8d8749-f5c9-4bbc-870a-bd919a44dec7" (UID: "de8d8749-f5c9-4bbc-870a-bd919a44dec7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:10.122438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.115046 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-dshm" pod "52740807-cfc4-4144-b4dc-14ef28aff6d6" (UID: "52740807-cfc4-4144-b4dc-14ef28aff6d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.122438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.115102 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8d8749-f5c9-4bbc-870a-bd919a44dec7-kube-api-access-gl4jf" pod "de8d8749-f5c9-4bbc-870a-bd919a44dec7" (UID: "de8d8749-f5c9-4bbc-870a-bd919a44dec7"). InnerVolumeSpecName "kube-api-access-gl4jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:10.122438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.115159 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52740807-cfc4-4144-b4dc-14ef28aff6d6-kube-api-access-x8jd8" pod "52740807-cfc4-4144-b4dc-14ef28aff6d6" (UID: "52740807-cfc4-4144-b4dc-14ef28aff6d6"). InnerVolumeSpecName "kube-api-access-x8jd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:10.122438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.119305 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-dshm" pod "de8d8749-f5c9-4bbc-870a-bd919a44dec7" (UID: "de8d8749-f5c9-4bbc-870a-bd919a44dec7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:10.209994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.209958 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.209994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.209996 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8jd8\" (UniqueName: \"kubernetes.io/projected/52740807-cfc4-4144-b4dc-14ef28aff6d6-kube-api-access-x8jd8\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.210232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.210010 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.210232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.210025 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52740807-cfc4-4144-b4dc-14ef28aff6d6-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.210232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.210038 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52740807-cfc4-4144-b4dc-14ef28aff6d6-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.210232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.210051 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.210232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.210062 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl4jf\" (UniqueName: \"kubernetes.io/projected/de8d8749-f5c9-4bbc-870a-bd919a44dec7-kube-api-access-gl4jf\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.210232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.210075 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de8d8749-f5c9-4bbc-870a-bd919a44dec7-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.210232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.210087 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de8d8749-f5c9-4bbc-870a-bd919a44dec7-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:10.303987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.303862 4952 generic.go:356] "Generic (PLEG): container finished" podID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerID="28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490" exitCode=0 Oct 29 22:42:10.303987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.303946 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" Oct 29 22:42:10.304205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.303944 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" event={"ID":"de8d8749-f5c9-4bbc-870a-bd919a44dec7","Type":"ContainerDied","Data":"28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490"} Oct 29 22:42:10.304205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.304064 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv" event={"ID":"de8d8749-f5c9-4bbc-870a-bd919a44dec7","Type":"ContainerDied","Data":"f75f3e71f54ed07733509cfb0250ece1197d4212034b6a219f14eff1e6d99d45"} Oct 29 22:42:10.304205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.304081 4952 scope.go:122] "RemoveContainer" containerID="28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490" Oct 29 22:42:10.305883 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.305858 4952 generic.go:356] "Generic (PLEG): container finished" podID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerID="64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3" exitCode=0 Oct 29 22:42:10.306021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.305896 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" event={"ID":"52740807-cfc4-4144-b4dc-14ef28aff6d6","Type":"ContainerDied","Data":"64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3"} Oct 29 22:42:10.306021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.305938 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" event={"ID":"52740807-cfc4-4144-b4dc-14ef28aff6d6","Type":"ContainerDied","Data":"2227b72d9a63ed11a2029da8319fb83c4f4ea05398915ef029da25b880017726"} Oct 29 22:42:10.306021 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.305900 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66" Oct 29 22:42:10.317674 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.317653 4952 scope.go:122] "RemoveContainer" containerID="28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490" Oct 29 22:42:10.318037 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:42:10.318015 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490\": container with ID starting with 28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490 not found: ID does not exist" containerID="28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490" Oct 29 22:42:10.318106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.318046 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490"} err="failed to get container status \"28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490\": rpc error: code = NotFound desc = could not find container \"28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490\": container with ID starting with 28e4d67911b63ef125d6902aba015893b7b0050212ba4ee07d8b0157c611e490 not found: ID does not exist" Oct 29 22:42:10.318106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.318066 4952 scope.go:122] "RemoveContainer" containerID="64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3" Oct 29 22:42:10.329321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.329297 4952 scope.go:122] "RemoveContainer" containerID="64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3" Oct 29 22:42:10.329700 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:42:10.329671 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3\": container with ID starting with 64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3 not found: ID does not exist" containerID="64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3" Oct 29 22:42:10.329830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.329704 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3"} err="failed to get container status \"64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3\": rpc error: code = NotFound desc = could not find container \"64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3\": container with ID starting with 64079b071fa7db7ba1f3a3cd4df52feb611fd5fb2b51854b44b7062e770c79f3 not found: ID does not exist" Oct 29 22:42:10.365080 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.365044 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv"] Oct 29 22:42:10.376239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.376203 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/simulator-w1-29c2f2-kserve-6bb95d8cd8-2vrbv"] Oct 29 22:42:10.411855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.411816 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66"] Oct 29 22:42:10.417395 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.417367 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-29c2f2/premium-w1-29c2f2-kserve-57bf997fc7-48j66"] Oct 29 22:42:10.648439 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.648397 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" path="/var/lib/kubelet/pods/52740807-cfc4-4144-b4dc-14ef28aff6d6/volumes" Oct 29 22:42:10.648994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:10.648972 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" path="/var/lib/kubelet/pods/de8d8749-f5c9-4bbc-870a-bd919a44dec7/volumes" Oct 29 22:42:11.467583 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:11.467531 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:12.038951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.038879 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" containerID="cri-o://b960d756db066895802d99883bac4c90296403f82d4d4d48c4f4383e98ea6aff" gracePeriod=45 Oct 29 22:42:12.112262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.112218 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": dial tcp 10.132.0.73:8000: connect: connection refused" Oct 29 22:42:12.325853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.325809 4952 generic.go:356] "Generic (PLEG): container finished" podID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerID="b960d756db066895802d99883bac4c90296403f82d4d4d48c4f4383e98ea6aff" exitCode=0 Oct 29 22:42:12.326015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.325876 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" event={"ID":"a012f50f-b410-4b94-96bb-1ea578bc090b","Type":"ContainerDied","Data":"b960d756db066895802d99883bac4c90296403f82d4d4d48c4f4383e98ea6aff"} Oct 29 22:42:12.326015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.325939 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" event={"ID":"a012f50f-b410-4b94-96bb-1ea578bc090b","Type":"ContainerDied","Data":"58aea1f27b7cc4844a34da365c4118230d0a702d889eed95a8ff85d656b22425"} Oct 29 22:42:12.326015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.325954 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58aea1f27b7cc4844a34da365c4118230d0a702d889eed95a8ff85d656b22425" Oct 29 22:42:12.333533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.333501 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:42:12.451140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.451097 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a012f50f-b410-4b94-96bb-1ea578bc090b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a012f50f-b410-4b94-96bb-1ea578bc090b-tls-certs\") pod \"a012f50f-b410-4b94-96bb-1ea578bc090b\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " Oct 29 22:42:12.451353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.451187 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a012f50f-b410-4b94-96bb-1ea578bc090b-kube-api-access-7jtf8\" (UniqueName: \"kubernetes.io/projected/a012f50f-b410-4b94-96bb-1ea578bc090b-kube-api-access-7jtf8\") pod \"a012f50f-b410-4b94-96bb-1ea578bc090b\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " Oct 29 22:42:12.451353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.451319 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-home\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-home\") pod \"a012f50f-b410-4b94-96bb-1ea578bc090b\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " Oct 29 22:42:12.451497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.451379 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-tmp-dir\") pod \"a012f50f-b410-4b94-96bb-1ea578bc090b\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " Oct 29 22:42:12.451497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.451485 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-dshm\") pod \"a012f50f-b410-4b94-96bb-1ea578bc090b\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " Oct 29 22:42:12.451612 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.451533 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-model-cache\") pod \"a012f50f-b410-4b94-96bb-1ea578bc090b\" (UID: \"a012f50f-b410-4b94-96bb-1ea578bc090b\") " Oct 29 22:42:12.453707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.452386 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-model-cache" pod "a012f50f-b410-4b94-96bb-1ea578bc090b" (UID: "a012f50f-b410-4b94-96bb-1ea578bc090b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.453707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.452840 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-home" pod "a012f50f-b410-4b94-96bb-1ea578bc090b" (UID: "a012f50f-b410-4b94-96bb-1ea578bc090b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.453707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.453640 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-tmp-dir" pod "a012f50f-b410-4b94-96bb-1ea578bc090b" (UID: "a012f50f-b410-4b94-96bb-1ea578bc090b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.456387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.456355 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a012f50f-b410-4b94-96bb-1ea578bc090b-kube-api-access-7jtf8" pod "a012f50f-b410-4b94-96bb-1ea578bc090b" (UID: "a012f50f-b410-4b94-96bb-1ea578bc090b"). InnerVolumeSpecName "kube-api-access-7jtf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:12.456515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.456451 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a012f50f-b410-4b94-96bb-1ea578bc090b-tls-certs" pod "a012f50f-b410-4b94-96bb-1ea578bc090b" (UID: "a012f50f-b410-4b94-96bb-1ea578bc090b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:12.457173 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.457148 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-dshm" pod "a012f50f-b410-4b94-96bb-1ea578bc090b" (UID: "a012f50f-b410-4b94-96bb-1ea578bc090b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:12.553782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.553688 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.553782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.553730 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.553782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.553745 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.553782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.553758 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a012f50f-b410-4b94-96bb-1ea578bc090b-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.553782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.553771 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtf8\" (UniqueName: \"kubernetes.io/projected/a012f50f-b410-4b94-96bb-1ea578bc090b-kube-api-access-7jtf8\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.554316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.553785 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a012f50f-b410-4b94-96bb-1ea578bc090b-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:12.959694 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:12.959622 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:13.423865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:13.417292 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65" Oct 29 22:42:13.507119 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:13.507077 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65"] Oct 29 22:42:13.532845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:13.532807 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cea94a/premium-w4-cea94a-kserve-59cc96949c-nvq65"] Oct 29 22:42:13.608339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:13.608303 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:13.793502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:13.793405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:14.257182 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:14.257147 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:14.648464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:14.648430 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" path="/var/lib/kubelet/pods/a012f50f-b410-4b94-96bb-1ea578bc090b/volumes" Oct 29 22:42:15.569570 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:15.568595 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:15.585179 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:42:15.585135 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:42:16.282404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:16.282370 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:42:16.298935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:16.298888 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:42:16.914190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:16.914154 4952 scope.go:122] "RemoveContainer" containerID="b960d756db066895802d99883bac4c90296403f82d4d4d48c4f4383e98ea6aff" Oct 29 22:42:17.203817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:17.200166 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:18.835031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:18.833864 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:20.569209 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:20.568983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:21.904751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:21.904709 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:22.145359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:22.145318 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:22.167255 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:22.167168 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd"] Oct 29 22:42:22.167785 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:22.167756 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" containerID="cri-o://6584e0e568a980d2bd5a42a2a970b8606d6ac856029c460e222598fdd6e4f45b" gracePeriod=30 Oct 29 22:42:22.341563 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:22.341523 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:22.522133 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:22.522044 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:24.194738 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:24.194701 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:25.891685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:25.891633 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:27.241648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.241613 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt"] Oct 29 22:42:27.406943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.404140 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:27.556378 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.556339 4952 generic.go:356] "Generic (PLEG): container finished" podID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerID="6584e0e568a980d2bd5a42a2a970b8606d6ac856029c460e222598fdd6e4f45b" exitCode=0 Oct 29 22:42:27.556822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.556799 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" event={"ID":"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1","Type":"ContainerDied","Data":"6584e0e568a980d2bd5a42a2a970b8606d6ac856029c460e222598fdd6e4f45b"} Oct 29 22:42:27.557003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.556989 4952 scope.go:122] "RemoveContainer" containerID="89c48aa2572127a93362586fe8b79e2589467ae0dd691ceecae21b7997b6cae6" Oct 29 22:42:27.659872 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.657215 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:42:27.709143 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709105 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-data\") pod \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " Oct 29 22:42:27.709143 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709148 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-envoy\") pod \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " Oct 29 22:42:27.709433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709270 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-certs\") pod \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " Oct 29 22:42:27.709433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709326 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istiod-ca-cert\") pod \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " Oct 29 22:42:27.709433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709366 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-socket\") pod \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " Oct 29 22:42:27.709433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709400 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-credential-socket\") pod \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " Oct 29 22:42:27.709433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709428 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-podinfo\") pod \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " Oct 29 22:42:27.709714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709444 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-data" pod "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" (UID: "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:27.709714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709490 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-token\" (UniqueName: \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-token\") pod \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " Oct 29 22:42:27.709714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709546 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-kube-api-access-6lmx9\" (UniqueName: \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-kube-api-access-6lmx9\") pod \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\" (UID: \"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1\") " Oct 29 22:42:27.709714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709631 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-certs" pod "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" (UID: "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:27.709897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709716 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-socket" pod "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" (UID: "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:27.709897 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709807 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istiod-ca-cert" pod "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" (UID: "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:27.710068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.709990 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-credential-socket" pod "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" (UID: "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:27.710238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.710213 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:27.710322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.710241 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:27.710322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.710257 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:27.710322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.710270 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:27.710322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.710286 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:27.711808 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.711781 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-envoy" pod "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" (UID: "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:27.712244 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.712203 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-token" pod "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" (UID: "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:27.712407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.712387 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-podinfo" pod "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" (UID: "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:42:27.712483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.712420 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-kube-api-access-6lmx9" pod "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" (UID: "79727b77-ad57-4b9d-b0fd-32a97fe5e4c1"). InnerVolumeSpecName "kube-api-access-6lmx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:27.825721 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.825688 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:27.825721 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.825723 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:27.826068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.825736 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:27.826068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.825749 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lmx9\" (UniqueName: \"kubernetes.io/projected/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1-kube-api-access-6lmx9\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:27.862378 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:27.862282 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:28.566648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:28.566608 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" event={"ID":"79727b77-ad57-4b9d-b0fd-32a97fe5e4c1","Type":"ContainerDied","Data":"402c70345fef1a6c70c1477a85a8cbb42a80f878fe27163d99b1d3e736cf8475"} Oct 29 22:42:28.566648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:28.566633 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd" Oct 29 22:42:28.567190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:28.566677 4952 scope.go:122] "RemoveContainer" containerID="6584e0e568a980d2bd5a42a2a970b8606d6ac856029c460e222598fdd6e4f45b" Oct 29 22:42:28.647250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:28.647220 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd"] Oct 29 22:42:28.656237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:28.656195 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-99e420-openshift-default-67664b8579-dhqvd"] Oct 29 22:42:29.544541 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:29.544501 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:30.646400 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:30.646364 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" path="/var/lib/kubelet/pods/79727b77-ad57-4b9d-b0fd-32a97fe5e4c1/volumes" Oct 29 22:42:31.389729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:31.389687 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:32.458749 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:32.458710 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:32.648765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:32.648728 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:33.457793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:33.457739 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:33.643483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:33.643446 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:34.822543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:34.822509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:35.011561 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:35.011515 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:35.463385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:35.463339 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:35.643455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:35.641717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:37.081920 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:37.081873 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:37.305380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:37.305341 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:39.096023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:39.095980 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg"] Oct 29 22:42:39.110267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:39.110228 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:39.335512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:39.335472 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:41.186690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:41.186652 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:41.408230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:41.408185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:42.313111 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.313070 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" podUID="9cf35311-cf34-4be6-847c-4088a59bf188" containerName="main" containerID="cri-o://5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9" gracePeriod=45 Oct 29 22:42:42.625129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.625094 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:42:42.702840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.702806 4952 generic.go:356] "Generic (PLEG): container finished" podID="9cf35311-cf34-4be6-847c-4088a59bf188" containerID="5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9" exitCode=0 Oct 29 22:42:42.703100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.703029 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" Oct 29 22:42:42.703100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.703050 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" event={"ID":"9cf35311-cf34-4be6-847c-4088a59bf188","Type":"ContainerDied","Data":"5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9"} Oct 29 22:42:42.703230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.703101 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt" event={"ID":"9cf35311-cf34-4be6-847c-4088a59bf188","Type":"ContainerDied","Data":"ae3f442bd206ea47c34b65bfb8ddf8bf591c9fc23580bbd6f1d76726c8b937be"} Oct 29 22:42:42.703230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.703124 4952 scope.go:122] "RemoveContainer" containerID="5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9" Oct 29 22:42:42.714859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.714833 4952 scope.go:122] "RemoveContainer" containerID="5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9" Oct 29 22:42:42.715221 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:42:42.715195 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9\": container with ID starting with 5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9 not found: ID does not exist" containerID="5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9" Oct 29 22:42:42.715316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.715233 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9"} err="failed to get container status \"5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9\": rpc error: code = NotFound desc = could not find container \"5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9\": container with ID starting with 5011aab9fec7ffeb2d0d2aa8066e3d17cf8071f5dfe19747e32ead8c7040b8a9 not found: ID does not exist" Oct 29 22:42:42.716695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.716665 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-tmp-dir\") pod \"9cf35311-cf34-4be6-847c-4088a59bf188\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " Oct 29 22:42:42.716823 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.716785 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-model-cache\") pod \"9cf35311-cf34-4be6-847c-4088a59bf188\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " Oct 29 22:42:42.716919 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.716857 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf35311-cf34-4be6-847c-4088a59bf188-kube-api-access-7s7zf\" (UniqueName: \"kubernetes.io/projected/9cf35311-cf34-4be6-847c-4088a59bf188-kube-api-access-7s7zf\") pod \"9cf35311-cf34-4be6-847c-4088a59bf188\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " Oct 29 22:42:42.716998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.716969 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-home\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-home\") pod \"9cf35311-cf34-4be6-847c-4088a59bf188\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " Oct 29 22:42:42.717088 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.717055 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-model-cache" pod "9cf35311-cf34-4be6-847c-4088a59bf188" (UID: "9cf35311-cf34-4be6-847c-4088a59bf188"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:42.717158 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.717077 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cf35311-cf34-4be6-847c-4088a59bf188-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf35311-cf34-4be6-847c-4088a59bf188-tls-certs\") pod \"9cf35311-cf34-4be6-847c-4088a59bf188\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " Oct 29 22:42:42.717227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.717194 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-tmp-dir" pod "9cf35311-cf34-4be6-847c-4088a59bf188" (UID: "9cf35311-cf34-4be6-847c-4088a59bf188"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:42.717227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.717200 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-dshm\") pod \"9cf35311-cf34-4be6-847c-4088a59bf188\" (UID: \"9cf35311-cf34-4be6-847c-4088a59bf188\") " Oct 29 22:42:42.717659 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.717441 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-home" pod "9cf35311-cf34-4be6-847c-4088a59bf188" (UID: "9cf35311-cf34-4be6-847c-4088a59bf188"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:42.718654 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.718202 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:42.718654 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.718231 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:42.718654 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.718245 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:42.719676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.719628 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-dshm" pod "9cf35311-cf34-4be6-847c-4088a59bf188" (UID: "9cf35311-cf34-4be6-847c-4088a59bf188"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:42.720016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.719991 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf35311-cf34-4be6-847c-4088a59bf188-kube-api-access-7s7zf" pod "9cf35311-cf34-4be6-847c-4088a59bf188" (UID: "9cf35311-cf34-4be6-847c-4088a59bf188"). InnerVolumeSpecName "kube-api-access-7s7zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:42.720688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.720665 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf35311-cf34-4be6-847c-4088a59bf188-tls-certs" pod "9cf35311-cf34-4be6-847c-4088a59bf188" (UID: "9cf35311-cf34-4be6-847c-4088a59bf188"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:42.819227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.819189 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf35311-cf34-4be6-847c-4088a59bf188-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:42.819227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.819223 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf35311-cf34-4be6-847c-4088a59bf188-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:42.819227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:42.819232 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s7zf\" (UniqueName: \"kubernetes.io/projected/9cf35311-cf34-4be6-847c-4088a59bf188-kube-api-access-7s7zf\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:43.049556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:43.049457 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:43.098817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:43.098779 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt"] Oct 29 22:42:43.102348 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:43.102314 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c06a2c/test-model-c06a2c-kserve-6c9f47f669-5t8zt"] Oct 29 22:42:44.268363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:44.268322 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:44.607677 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:44.607636 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:44.647865 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:44.647823 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf35311-cf34-4be6-847c-4088a59bf188" path="/var/lib/kubelet/pods/9cf35311-cf34-4be6-847c-4088a59bf188/volumes" Oct 29 22:42:45.533971 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:42:45.533927 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:42:46.474255 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:46.474219 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:48.269975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:48.269934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:49.489340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:49.489304 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:42:50.274523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:50.263473 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:50.600299 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:50.600258 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:52.183315 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:52.183275 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:53.936485 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:53.936449 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:54.130266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.130222 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" podUID="35243f45-e93c-48e5-aa76-f9cefdb58328" containerName="main" containerID="cri-o://7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1" gracePeriod=45 Oct 29 22:42:54.163468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.163427 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:54.455147 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.455112 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:42:54.524270 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.524187 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-model-cache\") pod \"35243f45-e93c-48e5-aa76-f9cefdb58328\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " Oct 29 22:42:54.524453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.524324 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-tmp-dir\") pod \"35243f45-e93c-48e5-aa76-f9cefdb58328\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " Oct 29 22:42:54.524453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.524423 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35243f45-e93c-48e5-aa76-f9cefdb58328-kube-api-access-dxjsq\" (UniqueName: \"kubernetes.io/projected/35243f45-e93c-48e5-aa76-f9cefdb58328-kube-api-access-dxjsq\") pod \"35243f45-e93c-48e5-aa76-f9cefdb58328\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " Oct 29 22:42:54.524586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.524505 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-model-cache" pod "35243f45-e93c-48e5-aa76-f9cefdb58328" (UID: "35243f45-e93c-48e5-aa76-f9cefdb58328"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:54.524695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.524668 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-tmp-dir" pod "35243f45-e93c-48e5-aa76-f9cefdb58328" (UID: "35243f45-e93c-48e5-aa76-f9cefdb58328"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:54.525095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.525067 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:54.525095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.525098 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:54.526879 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.526857 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35243f45-e93c-48e5-aa76-f9cefdb58328-kube-api-access-dxjsq" pod "35243f45-e93c-48e5-aa76-f9cefdb58328" (UID: "35243f45-e93c-48e5-aa76-f9cefdb58328"). InnerVolumeSpecName "kube-api-access-dxjsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:54.625700 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.625657 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-dshm\") pod \"35243f45-e93c-48e5-aa76-f9cefdb58328\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " Oct 29 22:42:54.625874 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.625705 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-home\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-home\") pod \"35243f45-e93c-48e5-aa76-f9cefdb58328\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " Oct 29 22:42:54.625874 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.625778 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35243f45-e93c-48e5-aa76-f9cefdb58328-tls-certs\" (UniqueName: \"kubernetes.io/secret/35243f45-e93c-48e5-aa76-f9cefdb58328-tls-certs\") pod \"35243f45-e93c-48e5-aa76-f9cefdb58328\" (UID: \"35243f45-e93c-48e5-aa76-f9cefdb58328\") " Oct 29 22:42:54.626092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.626015 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-home" pod "35243f45-e93c-48e5-aa76-f9cefdb58328" (UID: "35243f45-e93c-48e5-aa76-f9cefdb58328"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:54.626431 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.626408 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjsq\" (UniqueName: \"kubernetes.io/projected/35243f45-e93c-48e5-aa76-f9cefdb58328-kube-api-access-dxjsq\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:54.626540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.626434 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:54.628007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.627980 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-dshm" pod "35243f45-e93c-48e5-aa76-f9cefdb58328" (UID: "35243f45-e93c-48e5-aa76-f9cefdb58328"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:54.628210 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.628161 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35243f45-e93c-48e5-aa76-f9cefdb58328-tls-certs" pod "35243f45-e93c-48e5-aa76-f9cefdb58328" (UID: "35243f45-e93c-48e5-aa76-f9cefdb58328"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:42:54.727737 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.727686 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35243f45-e93c-48e5-aa76-f9cefdb58328-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:54.727737 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.727734 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35243f45-e93c-48e5-aa76-f9cefdb58328-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:42:54.804096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.804001 4952 generic.go:356] "Generic (PLEG): container finished" podID="35243f45-e93c-48e5-aa76-f9cefdb58328" containerID="7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1" exitCode=0 Oct 29 22:42:54.804096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.804056 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" Oct 29 22:42:54.804362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.804087 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" event={"ID":"35243f45-e93c-48e5-aa76-f9cefdb58328","Type":"ContainerDied","Data":"7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1"} Oct 29 22:42:54.804362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.804147 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg" event={"ID":"35243f45-e93c-48e5-aa76-f9cefdb58328","Type":"ContainerDied","Data":"e518edc4ad1751a4082ca97ca12bc567a9893d79a4dff7688ae93655ec224160"} Oct 29 22:42:54.804362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.804171 4952 scope.go:122] "RemoveContainer" containerID="7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1" Oct 29 22:42:54.815361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.815334 4952 scope.go:122] "RemoveContainer" containerID="7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1" Oct 29 22:42:54.815668 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:42:54.815640 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1\": container with ID starting with 7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1 not found: ID does not exist" containerID="7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1" Oct 29 22:42:54.815741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.815689 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1"} err="failed to get container status \"7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1\": rpc error: code = NotFound desc = could not find container \"7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1\": container with ID starting with 7f692682fdafab25641ad593c81d63d17e19e4327701750f2ee1378b4d13e3e1 not found: ID does not exist" Oct 29 22:42:54.870831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.870792 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg"] Oct 29 22:42:54.874298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:54.874266 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-90f46c/test-model-90f46c-kserve-855b4d4cb7-p5zjg"] Oct 29 22:42:55.682943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:55.682887 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:56.650476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:56.650434 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35243f45-e93c-48e5-aa76-f9cefdb58328" path="/var/lib/kubelet/pods/35243f45-e93c-48e5-aa76-f9cefdb58328/volumes" Oct 29 22:42:57.319115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:57.319063 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:57.934242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:57.934199 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:58.116803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:58.116767 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:58.552529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:58.552483 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:59.797930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:59.797781 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:59.979249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:42:59.979202 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:01.545150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:01.545102 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:03.274928 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:03.274863 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:04.951047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:04.950940 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:05.927845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:05.927803 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:06.157062 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:06.157024 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:06.364398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:06.364359 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:06.874605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:06.874568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:07.109619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:07.109578 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:08.902763 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:08.902703 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:10.637078 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:10.637037 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:11.776477 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:11.776425 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7"] Oct 29 22:43:11.811357 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:11.811310 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk"] Oct 29 22:43:11.817638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:11.817600 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx"] Oct 29 22:43:12.033832 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:12.033742 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:12.467366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:12.467328 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:13.756871 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:13.756825 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l"] Oct 29 22:43:13.757679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:13.757646 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" podUID="5a693545-a2b8-41c3-984c-74905ea0b217" containerName="maas-api" containerID="cri-o://5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1" gracePeriod=30 Oct 29 22:43:13.783327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:13.783293 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:13.963047 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:13.963011 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:14.483268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.483234 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:43:14.586465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.586428 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a693545-a2b8-41c3-984c-74905ea0b217-kube-api-access-4hfbj\" (UniqueName: \"kubernetes.io/projected/5a693545-a2b8-41c3-984c-74905ea0b217-kube-api-access-4hfbj\") pod \"5a693545-a2b8-41c3-984c-74905ea0b217\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " Oct 29 22:43:14.586712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.586528 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a693545-a2b8-41c3-984c-74905ea0b217-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a693545-a2b8-41c3-984c-74905ea0b217-openshift-service-ca\") pod \"5a693545-a2b8-41c3-984c-74905ea0b217\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " Oct 29 22:43:14.586798 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.586729 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-maas-api-tls\") pod \"5a693545-a2b8-41c3-984c-74905ea0b217\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " Oct 29 22:43:14.586862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.586816 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-metrics-certs\") pod \"5a693545-a2b8-41c3-984c-74905ea0b217\" (UID: \"5a693545-a2b8-41c3-984c-74905ea0b217\") " Oct 29 22:43:14.586970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.586940 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a693545-a2b8-41c3-984c-74905ea0b217-openshift-service-ca" pod "5a693545-a2b8-41c3-984c-74905ea0b217" (UID: "5a693545-a2b8-41c3-984c-74905ea0b217"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:14.587462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.587419 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a693545-a2b8-41c3-984c-74905ea0b217-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:14.589376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.589347 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-metrics-certs" pod "5a693545-a2b8-41c3-984c-74905ea0b217" (UID: "5a693545-a2b8-41c3-984c-74905ea0b217"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:14.589530 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.589506 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-maas-api-tls" pod "5a693545-a2b8-41c3-984c-74905ea0b217" (UID: "5a693545-a2b8-41c3-984c-74905ea0b217"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:14.589646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.589570 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a693545-a2b8-41c3-984c-74905ea0b217-kube-api-access-4hfbj" pod "5a693545-a2b8-41c3-984c-74905ea0b217" (UID: "5a693545-a2b8-41c3-984c-74905ea0b217"). InnerVolumeSpecName "kube-api-access-4hfbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:14.620386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.620302 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:14.688640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.688600 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:14.688640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.688635 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a693545-a2b8-41c3-984c-74905ea0b217-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:14.688640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.688645 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hfbj\" (UniqueName: \"kubernetes.io/projected/5a693545-a2b8-41c3-984c-74905ea0b217-kube-api-access-4hfbj\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:14.802337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.802300 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:14.965813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.965721 4952 generic.go:356] "Generic (PLEG): container finished" podID="5a693545-a2b8-41c3-984c-74905ea0b217" containerID="5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1" exitCode=0 Oct 29 22:43:14.965813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.965772 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" Oct 29 22:43:14.966069 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.965800 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" event={"ID":"5a693545-a2b8-41c3-984c-74905ea0b217","Type":"ContainerDied","Data":"5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1"} Oct 29 22:43:14.966069 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.965866 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l" event={"ID":"5a693545-a2b8-41c3-984c-74905ea0b217","Type":"ContainerDied","Data":"47729770a8e030ef1b1513429b0d60c8f58ccc57d229608bd457ed6c7c7c8730"} Oct 29 22:43:14.966069 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.965892 4952 scope.go:122] "RemoveContainer" containerID="5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1" Oct 29 22:43:14.979870 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.979844 4952 scope.go:122] "RemoveContainer" containerID="5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1" Oct 29 22:43:14.980224 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:43:14.980204 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1\": container with ID starting with 5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1 not found: ID does not exist" containerID="5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1" Oct 29 22:43:14.980288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:14.980234 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1"} err="failed to get container status \"5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1\": rpc error: code = NotFound desc = could not find container \"5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1\": container with ID starting with 5fa2f173bf8ebc94058e1c0d46ca04a4e6c444b7d1d6f27b8bb3476c623019a1 not found: ID does not exist" Oct 29 22:43:15.026255 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:15.026214 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l"] Oct 29 22:43:15.039639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:15.039601 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c06a2c-5c767bf57f-cds9l"] Oct 29 22:43:15.243532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:15.243440 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:15.529054 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:43:15.528960 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:43:16.205542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:16.205502 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:16.645418 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:16.645383 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a693545-a2b8-41c3-984c-74905ea0b217" path="/var/lib/kubelet/pods/5a693545-a2b8-41c3-984c-74905ea0b217/volumes" Oct 29 22:43:16.734470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:16.734435 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:17.026374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:17.026338 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:17.225533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:17.225495 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:17.424302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:17.424265 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:17.845655 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:17.845615 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:18.034860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:18.034822 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:18.496992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:18.491208 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:19.142457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:19.142423 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:19.317559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:19.317523 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:19.815496 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:19.815460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:21.033927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:21.033521 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:22.598427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:22.598389 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:22.769852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:22.769813 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:23.069388 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:23.069313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:24.590445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:24.590400 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:25.015807 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:25.015712 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:25.228123 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:25.228081 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:25.335779 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:25.335659 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw"] Oct 29 22:43:25.336747 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:25.336704 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" containerID="cri-o://4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598" gracePeriod=30 Oct 29 22:43:25.419607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:25.419563 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:26.504639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:26.504598 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:26.800945 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:26.800813 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" containerID="cri-o://9e4771a9e36082899bb37ab1561374b28d1a1ce57061cc8bc7315bc10badbb79" gracePeriod=45 Oct 29 22:43:26.834952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:26.834891 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" containerID="cri-o://74ea3a1ac3c74a4c2a6e4d3baaba1818578526215ac1c09b44b4434c5d219389" gracePeriod=45 Oct 29 22:43:26.839837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:26.839794 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" containerID="cri-o://d0fde757f51e9dd13c83c3bccb15e593a38323e79f01cc1c753ca69142371f06" gracePeriod=45 Oct 29 22:43:27.100818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.100461 4952 generic.go:356] "Generic (PLEG): container finished" podID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerID="74ea3a1ac3c74a4c2a6e4d3baaba1818578526215ac1c09b44b4434c5d219389" exitCode=0 Oct 29 22:43:27.100818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.100617 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" event={"ID":"b6a2de66-623a-4ede-af37-c683517ca3f4","Type":"ContainerDied","Data":"74ea3a1ac3c74a4c2a6e4d3baaba1818578526215ac1c09b44b4434c5d219389"} Oct 29 22:43:27.103607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.103541 4952 generic.go:356] "Generic (PLEG): container finished" podID="068d660a-401c-4b0a-9717-3a162259439a" containerID="9e4771a9e36082899bb37ab1561374b28d1a1ce57061cc8bc7315bc10badbb79" exitCode=0 Oct 29 22:43:27.104103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.103868 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" event={"ID":"068d660a-401c-4b0a-9717-3a162259439a","Type":"ContainerDied","Data":"9e4771a9e36082899bb37ab1561374b28d1a1ce57061cc8bc7315bc10badbb79"} Oct 29 22:43:27.106729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.106637 4952 generic.go:356] "Generic (PLEG): container finished" podID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerID="d0fde757f51e9dd13c83c3bccb15e593a38323e79f01cc1c753ca69142371f06" exitCode=0 Oct 29 22:43:27.106729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.106678 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" event={"ID":"e67b8901-4c7f-4f58-a38e-6da70147e552","Type":"ContainerDied","Data":"d0fde757f51e9dd13c83c3bccb15e593a38323e79f01cc1c753ca69142371f06"} Oct 29 22:43:27.196226 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.196200 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 22:43:27.204947 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.204917 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 22:43:27.207109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.207064 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251169 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e67b8901-4c7f-4f58-a38e-6da70147e552-tls-certs\" (UniqueName: \"kubernetes.io/secret/e67b8901-4c7f-4f58-a38e-6da70147e552-tls-certs\") pod \"e67b8901-4c7f-4f58-a38e-6da70147e552\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251208 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-home\") pod \"b6a2de66-623a-4ede-af37-c683517ca3f4\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251246 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67b8901-4c7f-4f58-a38e-6da70147e552-kube-api-access-78jz7\" (UniqueName: \"kubernetes.io/projected/e67b8901-4c7f-4f58-a38e-6da70147e552-kube-api-access-78jz7\") pod \"e67b8901-4c7f-4f58-a38e-6da70147e552\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251281 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-home\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-home\") pod \"e67b8901-4c7f-4f58-a38e-6da70147e552\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251310 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-tmp-dir\") pod \"e67b8901-4c7f-4f58-a38e-6da70147e552\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251329 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-model-cache\") pod \"e67b8901-4c7f-4f58-a38e-6da70147e552\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251410 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-dshm\") pod \"e67b8901-4c7f-4f58-a38e-6da70147e552\" (UID: \"e67b8901-4c7f-4f58-a38e-6da70147e552\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251434 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a2de66-623a-4ede-af37-c683517ca3f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2de66-623a-4ede-af37-c683517ca3f4-tls-certs\") pod \"b6a2de66-623a-4ede-af37-c683517ca3f4\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251474 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-dshm\") pod \"b6a2de66-623a-4ede-af37-c683517ca3f4\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251575 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a2de66-623a-4ede-af37-c683517ca3f4-kube-api-access-27xxg\" (UniqueName: \"kubernetes.io/projected/b6a2de66-623a-4ede-af37-c683517ca3f4-kube-api-access-27xxg\") pod \"b6a2de66-623a-4ede-af37-c683517ca3f4\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251600 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-model-cache\") pod \"b6a2de66-623a-4ede-af37-c683517ca3f4\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " Oct 29 22:43:27.251671 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.251631 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-tmp-dir\") pod \"b6a2de66-623a-4ede-af37-c683517ca3f4\" (UID: \"b6a2de66-623a-4ede-af37-c683517ca3f4\") " Oct 29 22:43:27.252479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.252358 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-model-cache" pod "e67b8901-4c7f-4f58-a38e-6da70147e552" (UID: "e67b8901-4c7f-4f58-a38e-6da70147e552"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.254549 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.253597 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-model-cache" pod "b6a2de66-623a-4ede-af37-c683517ca3f4" (UID: "b6a2de66-623a-4ede-af37-c683517ca3f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.254549 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.253827 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-home" pod "b6a2de66-623a-4ede-af37-c683517ca3f4" (UID: "b6a2de66-623a-4ede-af37-c683517ca3f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.254549 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.254194 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-home" pod "e67b8901-4c7f-4f58-a38e-6da70147e552" (UID: "e67b8901-4c7f-4f58-a38e-6da70147e552"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.254549 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.254498 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-tmp-dir" pod "e67b8901-4c7f-4f58-a38e-6da70147e552" (UID: "e67b8901-4c7f-4f58-a38e-6da70147e552"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.255282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.255259 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-tmp-dir" pod "b6a2de66-623a-4ede-af37-c683517ca3f4" (UID: "b6a2de66-623a-4ede-af37-c683517ca3f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.257150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.257119 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a2de66-623a-4ede-af37-c683517ca3f4-kube-api-access-27xxg" pod "b6a2de66-623a-4ede-af37-c683517ca3f4" (UID: "b6a2de66-623a-4ede-af37-c683517ca3f4"). InnerVolumeSpecName "kube-api-access-27xxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:27.257579 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.257552 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67b8901-4c7f-4f58-a38e-6da70147e552-tls-certs" pod "e67b8901-4c7f-4f58-a38e-6da70147e552" (UID: "e67b8901-4c7f-4f58-a38e-6da70147e552"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:27.258533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.258504 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-dshm" pod "e67b8901-4c7f-4f58-a38e-6da70147e552" (UID: "e67b8901-4c7f-4f58-a38e-6da70147e552"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.258969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.258894 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67b8901-4c7f-4f58-a38e-6da70147e552-kube-api-access-78jz7" pod "e67b8901-4c7f-4f58-a38e-6da70147e552" (UID: "e67b8901-4c7f-4f58-a38e-6da70147e552"). InnerVolumeSpecName "kube-api-access-78jz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:27.259162 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.259136 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-dshm" pod "b6a2de66-623a-4ede-af37-c683517ca3f4" (UID: "b6a2de66-623a-4ede-af37-c683517ca3f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.259486 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.259463 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a2de66-623a-4ede-af37-c683517ca3f4-tls-certs" pod "b6a2de66-623a-4ede-af37-c683517ca3f4" (UID: "b6a2de66-623a-4ede-af37-c683517ca3f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.356349 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-model-cache\") pod \"068d660a-401c-4b0a-9717-3a162259439a\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.356430 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-dshm\") pod \"068d660a-401c-4b0a-9717-3a162259439a\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.356467 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068d660a-401c-4b0a-9717-3a162259439a-kube-api-access-ld9tl\" (UniqueName: \"kubernetes.io/projected/068d660a-401c-4b0a-9717-3a162259439a-kube-api-access-ld9tl\") pod \"068d660a-401c-4b0a-9717-3a162259439a\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.356498 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-tmp-dir\") pod \"068d660a-401c-4b0a-9717-3a162259439a\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.356707 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-home\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-home\") pod \"068d660a-401c-4b0a-9717-3a162259439a\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.356750 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/068d660a-401c-4b0a-9717-3a162259439a-tls-certs\" (UniqueName: \"kubernetes.io/secret/068d660a-401c-4b0a-9717-3a162259439a-tls-certs\") pod \"068d660a-401c-4b0a-9717-3a162259439a\" (UID: \"068d660a-401c-4b0a-9717-3a162259439a\") " Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.357322 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.357344 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2de66-623a-4ede-af37-c683517ca3f4-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.357357 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.357371 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27xxg\" (UniqueName: \"kubernetes.io/projected/b6a2de66-623a-4ede-af37-c683517ca3f4-kube-api-access-27xxg\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.358679 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-tmp-dir" pod "068d660a-401c-4b0a-9717-3a162259439a" (UID: "068d660a-401c-4b0a-9717-3a162259439a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.358948 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-home" pod "068d660a-401c-4b0a-9717-3a162259439a" (UID: "068d660a-401c-4b0a-9717-3a162259439a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.359019 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-model-cache" pod "068d660a-401c-4b0a-9717-3a162259439a" (UID: "068d660a-401c-4b0a-9717-3a162259439a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.360927 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068d660a-401c-4b0a-9717-3a162259439a-kube-api-access-ld9tl" pod "068d660a-401c-4b0a-9717-3a162259439a" (UID: "068d660a-401c-4b0a-9717-3a162259439a"). InnerVolumeSpecName "kube-api-access-ld9tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:27.361967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.361021 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-dshm" pod "068d660a-401c-4b0a-9717-3a162259439a" (UID: "068d660a-401c-4b0a-9717-3a162259439a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:27.363221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.363068 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.363221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.363114 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.363221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.363130 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67b8901-4c7f-4f58-a38e-6da70147e552-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.363221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.363143 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2de66-623a-4ede-af37-c683517ca3f4-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.363221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.363158 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78jz7\" (UniqueName: \"kubernetes.io/projected/e67b8901-4c7f-4f58-a38e-6da70147e552-kube-api-access-78jz7\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.363221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.363172 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.363221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.363186 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.363221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.363201 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67b8901-4c7f-4f58-a38e-6da70147e552-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.370132 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.370075 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068d660a-401c-4b0a-9717-3a162259439a-tls-certs" pod "068d660a-401c-4b0a-9717-3a162259439a" (UID: "068d660a-401c-4b0a-9717-3a162259439a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:27.468310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.464662 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld9tl\" (UniqueName: \"kubernetes.io/projected/068d660a-401c-4b0a-9717-3a162259439a-kube-api-access-ld9tl\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.468310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.464706 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.468310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.464723 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.468310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.464736 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068d660a-401c-4b0a-9717-3a162259439a-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.468310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.464749 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.468310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.464762 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/068d660a-401c-4b0a-9717-3a162259439a-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:27.655840 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.655637 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:27.832308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.832200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:27.963172 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.963080 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp"] Oct 29 22:43:27.963358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:27.963285 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" containerID="cri-o://a9fb0b8801753859749f2a764bfef6a36e91d388c3bc2f92983331e5a2adb456" gracePeriod=30 Oct 29 22:43:28.056757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.056717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:28.088024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.087965 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:28.121997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.121963 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" Oct 29 22:43:28.122211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.121953 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk" event={"ID":"b6a2de66-623a-4ede-af37-c683517ca3f4","Type":"ContainerDied","Data":"1b90d8885ca2f297b3a01f23f81f72e95df841c604758382ff220b25575e7a4f"} Oct 29 22:43:28.122211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.122148 4952 scope.go:122] "RemoveContainer" containerID="74ea3a1ac3c74a4c2a6e4d3baaba1818578526215ac1c09b44b4434c5d219389" Oct 29 22:43:28.124595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.124559 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" event={"ID":"068d660a-401c-4b0a-9717-3a162259439a","Type":"ContainerDied","Data":"752870eaa5a458f758636b260f24f52aa46399407a6bce3335c89fa06e8e87bf"} Oct 29 22:43:28.124741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.124658 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7" Oct 29 22:43:28.127281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.127206 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" Oct 29 22:43:28.127281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.127246 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx" event={"ID":"e67b8901-4c7f-4f58-a38e-6da70147e552","Type":"ContainerDied","Data":"00055f66d9dc29e0f61b61da8b67b3cb78387baf10705935d735dae3e720c0a2"} Oct 29 22:43:28.138358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.138336 4952 scope.go:122] "RemoveContainer" containerID="9e4771a9e36082899bb37ab1561374b28d1a1ce57061cc8bc7315bc10badbb79" Oct 29 22:43:28.150946 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.150876 4952 scope.go:122] "RemoveContainer" containerID="d0fde757f51e9dd13c83c3bccb15e593a38323e79f01cc1c753ca69142371f06" Oct 29 22:43:28.189956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.189882 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:43:28.190142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.189974 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:43:28.200967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.200926 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk"] Oct 29 22:43:28.207469 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.207416 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/premium-w3-82db63-kserve-5fcc59f7f8-wqlqk"] Oct 29 22:43:28.254986 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.252329 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:28.286927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.284146 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx"] Oct 29 22:43:28.296930 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.295301 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/simulator-w3-82db63-kserve-85d4fd8c6c-4fmcx"] Oct 29 22:43:28.341058 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.341012 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7"] Oct 29 22:43:28.358510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.358467 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-82db63/distinct-2-w3-82db63-kserve-67976dd9c4-5dcg7"] Oct 29 22:43:28.645347 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.645312 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068d660a-401c-4b0a-9717-3a162259439a" path="/var/lib/kubelet/pods/068d660a-401c-4b0a-9717-3a162259439a/volumes" Oct 29 22:43:28.645965 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.645941 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" path="/var/lib/kubelet/pods/b6a2de66-623a-4ede-af37-c683517ca3f4/volumes" Oct 29 22:43:28.646560 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:28.646542 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" path="/var/lib/kubelet/pods/e67b8901-4c7f-4f58-a38e-6da70147e552/volumes" Oct 29 22:43:29.937929 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:29.936260 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:30.658350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.656151 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 22:43:30.747066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.747028 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-kube-api-access-b2s25\" (UniqueName: \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-kube-api-access-b2s25\") pod \"ff926c55-e2dc-49b5-9e20-ca7805198914\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " Oct 29 22:43:30.747283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.747093 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-envoy\") pod \"ff926c55-e2dc-49b5-9e20-ca7805198914\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " Oct 29 22:43:30.747283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.747225 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-socket\") pod \"ff926c55-e2dc-49b5-9e20-ca7805198914\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " Oct 29 22:43:30.747283 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.747260 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-token\" (UniqueName: \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-token\") pod \"ff926c55-e2dc-49b5-9e20-ca7805198914\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " Oct 29 22:43:30.748397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.747296 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff926c55-e2dc-49b5-9e20-ca7805198914-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff926c55-e2dc-49b5-9e20-ca7805198914-istiod-ca-cert\") pod \"ff926c55-e2dc-49b5-9e20-ca7805198914\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " Oct 29 22:43:30.748397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.747330 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-data\") pod \"ff926c55-e2dc-49b5-9e20-ca7805198914\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " Oct 29 22:43:30.748397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.747429 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-podinfo\") pod \"ff926c55-e2dc-49b5-9e20-ca7805198914\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " Oct 29 22:43:30.748397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.747484 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-certs\") pod \"ff926c55-e2dc-49b5-9e20-ca7805198914\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " Oct 29 22:43:30.748397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.747586 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-credential-socket\") pod \"ff926c55-e2dc-49b5-9e20-ca7805198914\" (UID: \"ff926c55-e2dc-49b5-9e20-ca7805198914\") " Oct 29 22:43:30.748397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.748356 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-credential-socket" pod "ff926c55-e2dc-49b5-9e20-ca7805198914" (UID: "ff926c55-e2dc-49b5-9e20-ca7805198914"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:30.748756 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.748578 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-certs" pod "ff926c55-e2dc-49b5-9e20-ca7805198914" (UID: "ff926c55-e2dc-49b5-9e20-ca7805198914"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:30.749458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.749206 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-socket" pod "ff926c55-e2dc-49b5-9e20-ca7805198914" (UID: "ff926c55-e2dc-49b5-9e20-ca7805198914"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:30.749458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.749384 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff926c55-e2dc-49b5-9e20-ca7805198914-istiod-ca-cert" pod "ff926c55-e2dc-49b5-9e20-ca7805198914" (UID: "ff926c55-e2dc-49b5-9e20-ca7805198914"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:30.749751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.749716 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-data" pod "ff926c55-e2dc-49b5-9e20-ca7805198914" (UID: "ff926c55-e2dc-49b5-9e20-ca7805198914"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:30.750299 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.750272 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-token" pod "ff926c55-e2dc-49b5-9e20-ca7805198914" (UID: "ff926c55-e2dc-49b5-9e20-ca7805198914"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:30.750768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.750746 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-kube-api-access-b2s25" pod "ff926c55-e2dc-49b5-9e20-ca7805198914" (UID: "ff926c55-e2dc-49b5-9e20-ca7805198914"). InnerVolumeSpecName "kube-api-access-b2s25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:30.750984 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.750951 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-podinfo" pod "ff926c55-e2dc-49b5-9e20-ca7805198914" (UID: "ff926c55-e2dc-49b5-9e20-ca7805198914"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:30.751793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.751765 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-envoy" pod "ff926c55-e2dc-49b5-9e20-ca7805198914" (UID: "ff926c55-e2dc-49b5-9e20-ca7805198914"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:30.849265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.849225 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.849265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.849259 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.849265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.849268 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff926c55-e2dc-49b5-9e20-ca7805198914-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.849265 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.849278 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.849559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.849289 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.849559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.849297 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.849559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.849306 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.849559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.849315 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2s25\" (UniqueName: \"kubernetes.io/projected/ff926c55-e2dc-49b5-9e20-ca7805198914-kube-api-access-b2s25\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.849559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.849323 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff926c55-e2dc-49b5-9e20-ca7805198914-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:30.892339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.892242 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:43:30.892339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:30.892316 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:43:31.161605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:31.161507 4952 generic.go:356] "Generic (PLEG): container finished" podID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerID="4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598" exitCode=0 Oct 29 22:43:31.161605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:31.161569 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" Oct 29 22:43:31.161605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:31.161587 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" event={"ID":"ff926c55-e2dc-49b5-9e20-ca7805198914","Type":"ContainerDied","Data":"4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598"} Oct 29 22:43:31.162248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:31.161645 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw" event={"ID":"ff926c55-e2dc-49b5-9e20-ca7805198914","Type":"ContainerDied","Data":"ec7f1f899addd9d0d4f9ac0151b559ebbeff9238243b948203f8ca5eca666e0d"} Oct 29 22:43:31.162248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:31.161673 4952 scope.go:122] "RemoveContainer" containerID="4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598" Oct 29 22:43:31.177010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:31.176981 4952 scope.go:122] "RemoveContainer" containerID="4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598" Oct 29 22:43:31.177389 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:43:31.177360 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598\": container with ID starting with 4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598 not found: ID does not exist" containerID="4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598" Oct 29 22:43:31.177458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:31.177404 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598"} err="failed to get container status \"4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598\": rpc error: code = NotFound desc = could not find container \"4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598\": container with ID starting with 4f2612a510643a8a7c28c3ebbdbb09433d1c5f5fc4f54e4fda53730eda97d598 not found: ID does not exist" Oct 29 22:43:31.230154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:31.230116 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw"] Oct 29 22:43:31.240795 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:31.240757 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29c2f2-openshift-default-95c856447-n8mlw"] Oct 29 22:43:31.699001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:31.698949 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:32.646553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:32.646519 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" path="/var/lib/kubelet/pods/ff926c55-e2dc-49b5-9e20-ca7805198914/volumes" Oct 29 22:43:33.185450 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.185406 4952 generic.go:356] "Generic (PLEG): container finished" podID="a6911000-63f9-412a-a097-ed410b7ce727" containerID="a9fb0b8801753859749f2a764bfef6a36e91d388c3bc2f92983331e5a2adb456" exitCode=0 Oct 29 22:43:33.185617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.185471 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" event={"ID":"a6911000-63f9-412a-a097-ed410b7ce727","Type":"ContainerDied","Data":"a9fb0b8801753859749f2a764bfef6a36e91d388c3bc2f92983331e5a2adb456"} Oct 29 22:43:33.268973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.268361 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 22:43:33.286867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.286828 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-istio-token\" (UniqueName: \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-istio-token\") pod \"a6911000-63f9-412a-a097-ed410b7ce727\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " Oct 29 22:43:33.287225 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.287197 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-kube-api-access-nczhk\" (UniqueName: \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-kube-api-access-nczhk\") pod \"a6911000-63f9-412a-a097-ed410b7ce727\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " Oct 29 22:43:33.287403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.287246 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-envoy\") pod \"a6911000-63f9-412a-a097-ed410b7ce727\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " Oct 29 22:43:33.287403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.287379 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6911000-63f9-412a-a097-ed410b7ce727-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6911000-63f9-412a-a097-ed410b7ce727-istiod-ca-cert\") pod \"a6911000-63f9-412a-a097-ed410b7ce727\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " Oct 29 22:43:33.287536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.287429 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-data\") pod \"a6911000-63f9-412a-a097-ed410b7ce727\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " Oct 29 22:43:33.287536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.287467 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-credential-socket\") pod \"a6911000-63f9-412a-a097-ed410b7ce727\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " Oct 29 22:43:33.287536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.287497 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6911000-63f9-412a-a097-ed410b7ce727-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6911000-63f9-412a-a097-ed410b7ce727-istio-podinfo\") pod \"a6911000-63f9-412a-a097-ed410b7ce727\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " Oct 29 22:43:33.287709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.287594 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-certs\") pod \"a6911000-63f9-412a-a097-ed410b7ce727\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " Oct 29 22:43:33.287709 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.287684 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-socket\") pod \"a6911000-63f9-412a-a097-ed410b7ce727\" (UID: \"a6911000-63f9-412a-a097-ed410b7ce727\") " Oct 29 22:43:33.288809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.288778 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:33.289379 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.289320 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-credential-socket" pod "a6911000-63f9-412a-a097-ed410b7ce727" (UID: "a6911000-63f9-412a-a097-ed410b7ce727"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:33.289697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.289673 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6911000-63f9-412a-a097-ed410b7ce727-istiod-ca-cert" pod "a6911000-63f9-412a-a097-ed410b7ce727" (UID: "a6911000-63f9-412a-a097-ed410b7ce727"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:33.289990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.289955 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-data" pod "a6911000-63f9-412a-a097-ed410b7ce727" (UID: "a6911000-63f9-412a-a097-ed410b7ce727"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:33.293201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.292309 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-certs" pod "a6911000-63f9-412a-a097-ed410b7ce727" (UID: "a6911000-63f9-412a-a097-ed410b7ce727"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:33.293201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.292466 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-envoy" pod "a6911000-63f9-412a-a097-ed410b7ce727" (UID: "a6911000-63f9-412a-a097-ed410b7ce727"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:33.293201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.292539 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-istio-token" pod "a6911000-63f9-412a-a097-ed410b7ce727" (UID: "a6911000-63f9-412a-a097-ed410b7ce727"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:33.293201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.292857 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-socket" pod "a6911000-63f9-412a-a097-ed410b7ce727" (UID: "a6911000-63f9-412a-a097-ed410b7ce727"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:33.303803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.303773 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6911000-63f9-412a-a097-ed410b7ce727-istio-podinfo" pod "a6911000-63f9-412a-a097-ed410b7ce727" (UID: "a6911000-63f9-412a-a097-ed410b7ce727"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:33.304097 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.304078 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-kube-api-access-nczhk" pod "a6911000-63f9-412a-a097-ed410b7ce727" (UID: "a6911000-63f9-412a-a097-ed410b7ce727"). InnerVolumeSpecName "kube-api-access-nczhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:33.389960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.389888 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.389960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.389966 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nczhk\" (UniqueName: \"kubernetes.io/projected/a6911000-63f9-412a-a097-ed410b7ce727-kube-api-access-nczhk\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.390177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.389980 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.390177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.389991 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6911000-63f9-412a-a097-ed410b7ce727-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.390177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.389999 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.390177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.390006 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.390177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.390016 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6911000-63f9-412a-a097-ed410b7ce727-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.390177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.390025 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:33.390177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:33.390033 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6911000-63f9-412a-a097-ed410b7ce727-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:34.200595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:34.199226 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" event={"ID":"a6911000-63f9-412a-a097-ed410b7ce727","Type":"ContainerDied","Data":"87d27279f3aa8cffa46291a4e40e108acb8d92f231e06be0d80d7aab1ecb5d3d"} Oct 29 22:43:34.200595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:34.199297 4952 scope.go:122] "RemoveContainer" containerID="a9fb0b8801753859749f2a764bfef6a36e91d388c3bc2f92983331e5a2adb456" Oct 29 22:43:34.200595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:34.199428 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp" Oct 29 22:43:34.262435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:34.262395 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:34.275174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:34.275140 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp"] Oct 29 22:43:34.328458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:34.328414 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cea94a-openshift-default-cd49c4f7f-49nqp"] Oct 29 22:43:34.440855 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:34.440816 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:34.652502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:34.652187 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6911000-63f9-412a-a097-ed410b7ce727" path="/var/lib/kubelet/pods/a6911000-63f9-412a-a097-ed410b7ce727/volumes" Oct 29 22:43:34.900394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:34.900351 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:36.506708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:36.506665 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:38.207954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:38.207887 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:39.819668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:39.818824 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:41.411878 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:41.411838 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:42.119415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:42.119378 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:42.982804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:42.982759 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:44.631938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:44.631883 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:45.527858 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:43:45.527819 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:43:46.230278 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:46.229051 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:47.927458 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:47.927400 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:49.611435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:49.611389 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:50.685776 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:50.685736 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:50.940554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:50.940457 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:50.993118 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:50.993085 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs"] Oct 29 22:43:50.993404 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:50.993369 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" podUID="bc7020b1-ea9e-450d-936b-40965f599b95" containerName="maas-api" containerID="cri-o://e9654c007c2c7c2bdf08e69be1f314d2354be3bd62cdceeab958c4116ace8757" gracePeriod=30 Oct 29 22:43:51.118376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.117071 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:51.306932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.304203 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:51.575519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.574863 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:51.578868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.577712 4952 generic.go:356] "Generic (PLEG): container finished" podID="bc7020b1-ea9e-450d-936b-40965f599b95" containerID="e9654c007c2c7c2bdf08e69be1f314d2354be3bd62cdceeab958c4116ace8757" exitCode=0 Oct 29 22:43:51.578868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.577981 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" event={"ID":"bc7020b1-ea9e-450d-936b-40965f599b95","Type":"ContainerDied","Data":"e9654c007c2c7c2bdf08e69be1f314d2354be3bd62cdceeab958c4116ace8757"} Oct 29 22:43:51.581800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.581478 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:43:51.672932 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.672821 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-metrics-certs\") pod \"bc7020b1-ea9e-450d-936b-40965f599b95\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " Oct 29 22:43:51.676962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.673351 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7020b1-ea9e-450d-936b-40965f599b95-kube-api-access-2hz4k\" (UniqueName: \"kubernetes.io/projected/bc7020b1-ea9e-450d-936b-40965f599b95-kube-api-access-2hz4k\") pod \"bc7020b1-ea9e-450d-936b-40965f599b95\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " Oct 29 22:43:51.676962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.673405 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-maas-api-tls\") pod \"bc7020b1-ea9e-450d-936b-40965f599b95\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " Oct 29 22:43:51.676962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.673441 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7020b1-ea9e-450d-936b-40965f599b95-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7020b1-ea9e-450d-936b-40965f599b95-openshift-service-ca\") pod \"bc7020b1-ea9e-450d-936b-40965f599b95\" (UID: \"bc7020b1-ea9e-450d-936b-40965f599b95\") " Oct 29 22:43:51.676962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.675124 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7020b1-ea9e-450d-936b-40965f599b95-openshift-service-ca" pod "bc7020b1-ea9e-450d-936b-40965f599b95" (UID: "bc7020b1-ea9e-450d-936b-40965f599b95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:51.685933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.684461 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7020b1-ea9e-450d-936b-40965f599b95-kube-api-access-2hz4k" pod "bc7020b1-ea9e-450d-936b-40965f599b95" (UID: "bc7020b1-ea9e-450d-936b-40965f599b95"). InnerVolumeSpecName "kube-api-access-2hz4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:51.685933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.684603 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-maas-api-tls" pod "bc7020b1-ea9e-450d-936b-40965f599b95" (UID: "bc7020b1-ea9e-450d-936b-40965f599b95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:51.707152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.707114 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-metrics-certs" pod "bc7020b1-ea9e-450d-936b-40965f599b95" (UID: "bc7020b1-ea9e-450d-936b-40965f599b95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:51.750653 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.750535 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:51.786647 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.786603 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:51.786647 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.786643 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hz4k\" (UniqueName: \"kubernetes.io/projected/bc7020b1-ea9e-450d-936b-40965f599b95-kube-api-access-2hz4k\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:51.787023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.786660 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7020b1-ea9e-450d-936b-40965f599b95-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:51.787023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.786677 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7020b1-ea9e-450d-936b-40965f599b95-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:51.990838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:51.990730 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:52.162882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:52.162840 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:52.597197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:52.597156 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" event={"ID":"bc7020b1-ea9e-450d-936b-40965f599b95","Type":"ContainerDied","Data":"f4d1d4a26ec6d037195b9e6521121d146ed51c142501d2b72b1cac8879c10625"} Oct 29 22:43:52.597410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:52.597209 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs" Oct 29 22:43:52.597410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:52.597247 4952 scope.go:122] "RemoveContainer" containerID="e9654c007c2c7c2bdf08e69be1f314d2354be3bd62cdceeab958c4116ace8757" Oct 29 22:43:52.618536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:52.618505 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:52.658137 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:52.658103 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs"] Oct 29 22:43:52.670263 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:52.670229 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90f46c-74f697b8b9-6vzxs"] Oct 29 22:43:53.854402 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:53.854366 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94"] Oct 29 22:43:53.854846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:53.854595 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" podUID="64fa5b64-54c9-4cb9-aa39-771246749378" containerName="discovery" containerID="cri-o://df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de" gracePeriod=30 Oct 29 22:43:54.027250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.027209 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:54.159090 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.159060 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 22:43:54.217437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.217398 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-kube-api-access-lkdfz\" (UniqueName: \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-kube-api-access-lkdfz\") pod \"64fa5b64-54c9-4cb9-aa39-771246749378\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " Oct 29 22:43:54.217639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.217490 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-istio-token\" (UniqueName: \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-istio-token\") pod \"64fa5b64-54c9-4cb9-aa39-771246749378\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " Oct 29 22:43:54.217639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.217531 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-kubeconfig\") pod \"64fa5b64-54c9-4cb9-aa39-771246749378\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " Oct 29 22:43:54.217639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.217631 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64fa5b64-54c9-4cb9-aa39-771246749378-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fa5b64-54c9-4cb9-aa39-771246749378-local-certs\") pod \"64fa5b64-54c9-4cb9-aa39-771246749378\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " Oct 29 22:43:54.217806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.217694 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-cacerts\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-cacerts\") pod \"64fa5b64-54c9-4cb9-aa39-771246749378\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " Oct 29 22:43:54.217806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.217746 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-ca-configmap\") pod \"64fa5b64-54c9-4cb9-aa39-771246749378\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " Oct 29 22:43:54.217969 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.217864 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-dns-cert\") pod \"64fa5b64-54c9-4cb9-aa39-771246749378\" (UID: \"64fa5b64-54c9-4cb9-aa39-771246749378\") " Oct 29 22:43:54.218760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.218707 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-ca-configmap" pod "64fa5b64-54c9-4cb9-aa39-771246749378" (UID: "64fa5b64-54c9-4cb9-aa39-771246749378"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:54.220469 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.220430 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-istio-token" pod "64fa5b64-54c9-4cb9-aa39-771246749378" (UID: "64fa5b64-54c9-4cb9-aa39-771246749378"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:54.221140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.221111 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-dns-cert" pod "64fa5b64-54c9-4cb9-aa39-771246749378" (UID: "64fa5b64-54c9-4cb9-aa39-771246749378"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:54.221614 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.221583 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-cacerts" pod "64fa5b64-54c9-4cb9-aa39-771246749378" (UID: "64fa5b64-54c9-4cb9-aa39-771246749378"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:54.221780 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.221744 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fa5b64-54c9-4cb9-aa39-771246749378-local-certs" pod "64fa5b64-54c9-4cb9-aa39-771246749378" (UID: "64fa5b64-54c9-4cb9-aa39-771246749378"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:54.222013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.221985 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-kube-api-access-lkdfz" pod "64fa5b64-54c9-4cb9-aa39-771246749378" (UID: "64fa5b64-54c9-4cb9-aa39-771246749378"). InnerVolumeSpecName "kube-api-access-lkdfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:54.222122 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.222044 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-kubeconfig" pod "64fa5b64-54c9-4cb9-aa39-771246749378" (UID: "64fa5b64-54c9-4cb9-aa39-771246749378"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:43:54.320281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.320231 4952 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-dns-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:54.320281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.320280 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkdfz\" (UniqueName: \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-kube-api-access-lkdfz\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:54.320639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.320296 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fa5b64-54c9-4cb9-aa39-771246749378-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:54.320639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.320310 4952 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-istio-kubeconfig\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:54.320639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.320323 4952 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fa5b64-54c9-4cb9-aa39-771246749378-local-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:54.320639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.320336 4952 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64fa5b64-54c9-4cb9-aa39-771246749378-cacerts\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:54.320639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.320349 4952 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64fa5b64-54c9-4cb9-aa39-771246749378-istio-csr-ca-configmap\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:43:54.612768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.612734 4952 generic.go:356] "Generic (PLEG): container finished" podID="64fa5b64-54c9-4cb9-aa39-771246749378" containerID="df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de" exitCode=0 Oct 29 22:43:54.613034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.612778 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" Oct 29 22:43:54.613034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.612817 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" event={"ID":"64fa5b64-54c9-4cb9-aa39-771246749378","Type":"ContainerDied","Data":"df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de"} Oct 29 22:43:54.613034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.612875 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94" event={"ID":"64fa5b64-54c9-4cb9-aa39-771246749378","Type":"ContainerDied","Data":"e434a9ecdd70d659e8e88b88062175239d778a8daa9356a5c422d9d3c7e5fd2d"} Oct 29 22:43:54.613034 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.612901 4952 scope.go:122] "RemoveContainer" containerID="df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de" Oct 29 22:43:54.627696 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.627670 4952 scope.go:122] "RemoveContainer" containerID="df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de" Oct 29 22:43:54.628083 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:43:54.628051 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de\": container with ID starting with df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de not found: ID does not exist" containerID="df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de" Oct 29 22:43:54.628162 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.628094 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de"} err="failed to get container status \"df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de\": rpc error: code = NotFound desc = could not find container \"df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de\": container with ID starting with df11d6dc7a4df8d9948450ed52c38e4df9bf9b2d749ff06dd68560a5c13cd6de not found: ID does not exist" Oct 29 22:43:54.645660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.645622 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7020b1-ea9e-450d-936b-40965f599b95" path="/var/lib/kubelet/pods/bc7020b1-ea9e-450d-936b-40965f599b95/volumes" Oct 29 22:43:54.675937 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.675864 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94"] Oct 29 22:43:54.687735 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:54.687700 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8rs94"] Oct 29 22:43:55.513553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:55.513506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:56.650276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:56.650235 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64fa5b64-54c9-4cb9-aa39-771246749378" path="/var/lib/kubelet/pods/64fa5b64-54c9-4cb9-aa39-771246749378/volumes" Oct 29 22:43:57.104312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:57.104275 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:58.665224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:58.665117 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:59.161981 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:59.161950 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:59.338755 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:59.338716 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:59.810637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:59.808314 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:59.975215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:43:59.975165 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:00.083939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:00.083869 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g"] Oct 29 22:44:00.084196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:00.084166 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" containerID="cri-o://2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b" gracePeriod=30 Oct 29 22:44:00.172453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:00.172413 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:00.345483 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:00.345343 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:02.116129 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:02.116091 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:03.473092 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:03.473042 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:44:03.863954 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:03.863888 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:05.385216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.385183 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:44:05.479633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.479589 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-credential-socket\") pod \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " Oct 29 22:44:05.479952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.479892 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-credential-socket" pod "d84b5278-d243-470c-abf8-e6e8e9aa3e62" (UID: "d84b5278-d243-470c-abf8-e6e8e9aa3e62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:05.479952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.479926 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istiod-ca-cert\") pod \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " Oct 29 22:44:05.480230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.480090 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-socket\") pod \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " Oct 29 22:44:05.480230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.480204 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-kube-api-access-zm9fc\" (UniqueName: \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-kube-api-access-zm9fc\") pod \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " Oct 29 22:44:05.480397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.480347 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istiod-ca-cert" pod "d84b5278-d243-470c-abf8-e6e8e9aa3e62" (UID: "d84b5278-d243-470c-abf8-e6e8e9aa3e62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:05.480397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.480352 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-socket" pod "d84b5278-d243-470c-abf8-e6e8e9aa3e62" (UID: "d84b5278-d243-470c-abf8-e6e8e9aa3e62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:05.480397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.480381 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-envoy\") pod \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " Oct 29 22:44:05.480576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.480449 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-token\" (UniqueName: \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-token\") pod \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " Oct 29 22:44:05.480576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.480486 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-certs\") pod \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " Oct 29 22:44:05.480576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.480559 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-data\") pod \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " Oct 29 22:44:05.480740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.480621 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-podinfo\") pod \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\" (UID: \"d84b5278-d243-470c-abf8-e6e8e9aa3e62\") " Oct 29 22:44:05.481186 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.481161 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.481335 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.481188 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.481335 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.481202 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.481335 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.481293 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-data" pod "d84b5278-d243-470c-abf8-e6e8e9aa3e62" (UID: "d84b5278-d243-470c-abf8-e6e8e9aa3e62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:05.483809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.481381 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-certs" pod "d84b5278-d243-470c-abf8-e6e8e9aa3e62" (UID: "d84b5278-d243-470c-abf8-e6e8e9aa3e62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:05.483809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.483512 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-podinfo" pod "d84b5278-d243-470c-abf8-e6e8e9aa3e62" (UID: "d84b5278-d243-470c-abf8-e6e8e9aa3e62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:05.483809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.483560 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-envoy" pod "d84b5278-d243-470c-abf8-e6e8e9aa3e62" (UID: "d84b5278-d243-470c-abf8-e6e8e9aa3e62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:05.483809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.483655 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-token" pod "d84b5278-d243-470c-abf8-e6e8e9aa3e62" (UID: "d84b5278-d243-470c-abf8-e6e8e9aa3e62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:05.485767 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.485743 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-kube-api-access-zm9fc" pod "d84b5278-d243-470c-abf8-e6e8e9aa3e62" (UID: "d84b5278-d243-470c-abf8-e6e8e9aa3e62"). InnerVolumeSpecName "kube-api-access-zm9fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:05.582308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.582268 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.582308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.582303 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.582308 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.582316 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.582558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.582328 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.582558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.582339 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84b5278-d243-470c-abf8-e6e8e9aa3e62-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.582558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.582348 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm9fc\" (UniqueName: \"kubernetes.io/projected/d84b5278-d243-470c-abf8-e6e8e9aa3e62-kube-api-access-zm9fc\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:05.724286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.724198 4952 generic.go:356] "Generic (PLEG): container finished" podID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerID="2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b" exitCode=0 Oct 29 22:44:05.724286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.724263 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" Oct 29 22:44:05.724543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.724279 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" event={"ID":"d84b5278-d243-470c-abf8-e6e8e9aa3e62","Type":"ContainerDied","Data":"2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b"} Oct 29 22:44:05.724543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.724359 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g" event={"ID":"d84b5278-d243-470c-abf8-e6e8e9aa3e62","Type":"ContainerDied","Data":"51f100f67753f71b4f44532487092d2353f74b75019df34d6531450bd108613a"} Oct 29 22:44:05.724543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.724392 4952 scope.go:122] "RemoveContainer" containerID="2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b" Oct 29 22:44:05.737270 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.737248 4952 scope.go:122] "RemoveContainer" containerID="52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21" Oct 29 22:44:05.751199 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.751175 4952 scope.go:122] "RemoveContainer" containerID="2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b" Oct 29 22:44:05.751535 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:44:05.751508 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b\": container with ID starting with 2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b not found: ID does not exist" containerID="2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b" Oct 29 22:44:05.751652 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.751548 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b"} err="failed to get container status \"2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b\": rpc error: code = NotFound desc = could not find container \"2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b\": container with ID starting with 2bfb94232524567d387e112200efc421b5fb2ebdf7579d7e615111de8739630b not found: ID does not exist" Oct 29 22:44:05.751652 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.751572 4952 scope.go:122] "RemoveContainer" containerID="52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21" Oct 29 22:44:05.751962 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:44:05.751941 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21\": container with ID starting with 52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21 not found: ID does not exist" containerID="52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21" Oct 29 22:44:05.752024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.751971 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21"} err="failed to get container status \"52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21\": rpc error: code = NotFound desc = could not find container \"52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21\": container with ID starting with 52b54f1c8410293bc7ded80fb676a4a3dd5a7580caf55559da3f2f250d5e3b21 not found: ID does not exist" Oct 29 22:44:05.784872 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.784840 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g"] Oct 29 22:44:05.796321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.796291 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-90f46c-openshift-default-864497cdf-rsm2g"] Oct 29 22:44:05.839465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:05.839427 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:06.644700 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:06.644657 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" path="/var/lib/kubelet/pods/d84b5278-d243-470c-abf8-e6e8e9aa3e62/volumes" Oct 29 22:44:07.426462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:07.426421 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:09.017950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:09.017890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:10.616962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:10.616834 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:11.503229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:11.503179 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:11.684191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:11.684150 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:12.338813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:12.338761 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:13.880192 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:13.880158 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:14.050842 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:14.050807 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:15.527084 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:44:15.527041 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:44:15.586424 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:15.586389 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:17.141198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:17.141155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:18.808345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:18.808285 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:20.335289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:20.335241 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:21.890553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:21.890521 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:23.299875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:23.299837 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:23.797724 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:23.797680 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:23.969959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:23.969920 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:24.472363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:24.472322 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:25.722168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:25.722127 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:26.858973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:26.858929 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:44:27.452316 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:27.452277 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:29.169756 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:29.169720 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:30.704851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:30.704812 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:31.373750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:31.373713 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:31.541652 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:31.541615 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:31.678845 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:31.678752 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7"] Oct 29 22:44:31.679069 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:31.679039 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" containerID="cri-o://3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245" gracePeriod=30 Oct 29 22:44:31.778883 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:31.778836 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:32.302953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:32.302881 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:33.542951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.542897 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp"] Oct 29 22:44:33.543768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.543719 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" podUID="5d367b46-cebf-42bf-93d4-a5c558ec1448" containerName="maas-api" containerID="cri-o://4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad" gracePeriod=30 Oct 29 22:44:33.646019 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.645967 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:33.919964 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.918399 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:44:33.976782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.976519 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d367b46-cebf-42bf-93d4-a5c558ec1448-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d367b46-cebf-42bf-93d4-a5c558ec1448-openshift-service-ca\") pod \"5d367b46-cebf-42bf-93d4-a5c558ec1448\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " Oct 29 22:44:33.976782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.976594 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-metrics-certs\") pod \"5d367b46-cebf-42bf-93d4-a5c558ec1448\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " Oct 29 22:44:33.976782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.976715 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d367b46-cebf-42bf-93d4-a5c558ec1448-kube-api-access-97qq4\" (UniqueName: \"kubernetes.io/projected/5d367b46-cebf-42bf-93d4-a5c558ec1448-kube-api-access-97qq4\") pod \"5d367b46-cebf-42bf-93d4-a5c558ec1448\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " Oct 29 22:44:33.976782 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.976794 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-maas-api-tls\") pod \"5d367b46-cebf-42bf-93d4-a5c558ec1448\" (UID: \"5d367b46-cebf-42bf-93d4-a5c558ec1448\") " Oct 29 22:44:34.003148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.980593 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d367b46-cebf-42bf-93d4-a5c558ec1448-openshift-service-ca" pod "5d367b46-cebf-42bf-93d4-a5c558ec1448" (UID: "5d367b46-cebf-42bf-93d4-a5c558ec1448"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:34.003148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.983341 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-metrics-certs" pod "5d367b46-cebf-42bf-93d4-a5c558ec1448" (UID: "5d367b46-cebf-42bf-93d4-a5c558ec1448"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:34.003148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.988729 4952 generic.go:356] "Generic (PLEG): container finished" podID="5d367b46-cebf-42bf-93d4-a5c558ec1448" containerID="4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad" exitCode=0 Oct 29 22:44:34.003148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.988797 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" event={"ID":"5d367b46-cebf-42bf-93d4-a5c558ec1448","Type":"ContainerDied","Data":"4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad"} Oct 29 22:44:34.003148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.988843 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" event={"ID":"5d367b46-cebf-42bf-93d4-a5c558ec1448","Type":"ContainerDied","Data":"f5f9acb4e7df0e47b3de7d5d17572d9054ba8ceabaf954c956fb8126c9826a55"} Oct 29 22:44:34.003148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.988871 4952 scope.go:122] "RemoveContainer" containerID="4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad" Oct 29 22:44:34.003148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.989048 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp" Oct 29 22:44:34.003148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:33.996140 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d367b46-cebf-42bf-93d4-a5c558ec1448-kube-api-access-97qq4" pod "5d367b46-cebf-42bf-93d4-a5c558ec1448" (UID: "5d367b46-cebf-42bf-93d4-a5c558ec1448"). InnerVolumeSpecName "kube-api-access-97qq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:34.008764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.008738 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-maas-api-tls" pod "5d367b46-cebf-42bf-93d4-a5c558ec1448" (UID: "5d367b46-cebf-42bf-93d4-a5c558ec1448"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:34.019718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.019617 4952 scope.go:122] "RemoveContainer" containerID="4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad" Oct 29 22:44:34.020113 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:44:34.020082 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad\": container with ID starting with 4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad not found: ID does not exist" containerID="4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad" Oct 29 22:44:34.020234 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.020118 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad"} err="failed to get container status \"4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad\": rpc error: code = NotFound desc = could not find container \"4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad\": container with ID starting with 4aaf4361489e2ca98a0a28c094a672c463ea2cc99510cdeed32155b083d9c5ad not found: ID does not exist" Oct 29 22:44:34.078383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.078343 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d367b46-cebf-42bf-93d4-a5c558ec1448-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:34.078383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.078378 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:34.078594 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.078393 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97qq4\" (UniqueName: \"kubernetes.io/projected/5d367b46-cebf-42bf-93d4-a5c558ec1448-kube-api-access-97qq4\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:34.078594 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.078406 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d367b46-cebf-42bf-93d4-a5c558ec1448-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:34.183537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.183457 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:34.357349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.357315 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp"] Oct 29 22:44:34.363164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.363130 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b7c2d5-686bbd4859-tmkdp"] Oct 29 22:44:34.414019 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.413983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:34.644042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.644009 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d367b46-cebf-42bf-93d4-a5c558ec1448" path="/var/lib/kubelet/pods/5d367b46-cebf-42bf-93d4-a5c558ec1448/volumes" Oct 29 22:44:34.830707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:34.830664 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:36.111231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:36.111191 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:36.954668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:36.954601 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 22:44:37.015486 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.015452 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-token\" (UniqueName: \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-token\") pod \"e971f8a4-abc5-4235-8abb-4f26d225e35b\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " Oct 29 22:44:37.015708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.015504 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-credential-socket\") pod \"e971f8a4-abc5-4235-8abb-4f26d225e35b\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " Oct 29 22:44:37.015708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.015614 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-data\") pod \"e971f8a4-abc5-4235-8abb-4f26d225e35b\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " Oct 29 22:44:37.015708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.015658 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-certs\") pod \"e971f8a4-abc5-4235-8abb-4f26d225e35b\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " Oct 29 22:44:37.015895 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.015710 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-podinfo\") pod \"e971f8a4-abc5-4235-8abb-4f26d225e35b\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " Oct 29 22:44:37.015895 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.015793 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-kube-api-access-rtllm\" (UniqueName: \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-kube-api-access-rtllm\") pod \"e971f8a4-abc5-4235-8abb-4f26d225e35b\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " Oct 29 22:44:37.015895 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.015845 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-socket\") pod \"e971f8a4-abc5-4235-8abb-4f26d225e35b\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " Oct 29 22:44:37.016102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.015939 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-envoy\") pod \"e971f8a4-abc5-4235-8abb-4f26d225e35b\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " Oct 29 22:44:37.016102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.015973 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-credential-socket" pod "e971f8a4-abc5-4235-8abb-4f26d225e35b" (UID: "e971f8a4-abc5-4235-8abb-4f26d225e35b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:37.016102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.015983 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e971f8a4-abc5-4235-8abb-4f26d225e35b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e971f8a4-abc5-4235-8abb-4f26d225e35b-istiod-ca-cert\") pod \"e971f8a4-abc5-4235-8abb-4f26d225e35b\" (UID: \"e971f8a4-abc5-4235-8abb-4f26d225e35b\") " Oct 29 22:44:37.016468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.016447 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.016755 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.016737 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e971f8a4-abc5-4235-8abb-4f26d225e35b-istiod-ca-cert" pod "e971f8a4-abc5-4235-8abb-4f26d225e35b" (UID: "e971f8a4-abc5-4235-8abb-4f26d225e35b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:37.017219 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.017182 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-data" pod "e971f8a4-abc5-4235-8abb-4f26d225e35b" (UID: "e971f8a4-abc5-4235-8abb-4f26d225e35b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:37.017219 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.017199 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-socket" pod "e971f8a4-abc5-4235-8abb-4f26d225e35b" (UID: "e971f8a4-abc5-4235-8abb-4f26d225e35b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:37.017638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.017617 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-certs" pod "e971f8a4-abc5-4235-8abb-4f26d225e35b" (UID: "e971f8a4-abc5-4235-8abb-4f26d225e35b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:37.029985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.019964 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-token" pod "e971f8a4-abc5-4235-8abb-4f26d225e35b" (UID: "e971f8a4-abc5-4235-8abb-4f26d225e35b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:37.035495 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.035415 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-podinfo" pod "e971f8a4-abc5-4235-8abb-4f26d225e35b" (UID: "e971f8a4-abc5-4235-8abb-4f26d225e35b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:37.043000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.040566 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-envoy" pod "e971f8a4-abc5-4235-8abb-4f26d225e35b" (UID: "e971f8a4-abc5-4235-8abb-4f26d225e35b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:37.043000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.040760 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-kube-api-access-rtllm" pod "e971f8a4-abc5-4235-8abb-4f26d225e35b" (UID: "e971f8a4-abc5-4235-8abb-4f26d225e35b"). InnerVolumeSpecName "kube-api-access-rtllm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:37.053707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.053667 4952 generic.go:356] "Generic (PLEG): container finished" podID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerID="3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245" exitCode=0 Oct 29 22:44:37.053839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.053764 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" event={"ID":"e971f8a4-abc5-4235-8abb-4f26d225e35b","Type":"ContainerDied","Data":"3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245"} Oct 29 22:44:37.053938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.053850 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" event={"ID":"e971f8a4-abc5-4235-8abb-4f26d225e35b","Type":"ContainerDied","Data":"3d636190085d1c2eb8eab6b3ace8af2325aa9253bd5b86f68c618300200df8a7"} Oct 29 22:44:37.053938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.053891 4952 scope.go:122] "RemoveContainer" containerID="3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245" Oct 29 22:44:37.054178 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.054114 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7" Oct 29 22:44:37.083247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.083095 4952 scope.go:122] "RemoveContainer" containerID="3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245" Oct 29 22:44:37.083521 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:44:37.083489 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245\": container with ID starting with 3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245 not found: ID does not exist" containerID="3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245" Oct 29 22:44:37.083627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.083529 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245"} err="failed to get container status \"3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245\": rpc error: code = NotFound desc = could not find container \"3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245\": container with ID starting with 3d74a11bbfe7b8b4768b0b38b9029582c2868f28dbf8e8ad2a2f68e9b67f0245 not found: ID does not exist" Oct 29 22:44:37.118016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.117986 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.118016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.118014 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.118016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.118023 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.118470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.118032 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.118470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.118041 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtllm\" (UniqueName: \"kubernetes.io/projected/e971f8a4-abc5-4235-8abb-4f26d225e35b-kube-api-access-rtllm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.118470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.118051 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.118470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.118059 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e971f8a4-abc5-4235-8abb-4f26d225e35b-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.118470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.118067 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e971f8a4-abc5-4235-8abb-4f26d225e35b-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:44:37.123823 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.123796 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7"] Oct 29 22:44:37.131412 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.131373 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82db63-openshift-default-59dc95d95c-55nh7"] Oct 29 22:44:37.655309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:37.655267 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:38.646938 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:38.646885 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" path="/var/lib/kubelet/pods/e971f8a4-abc5-4235-8abb-4f26d225e35b/volumes" Oct 29 22:44:39.207206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:39.207163 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:40.811683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:40.811627 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:41.942860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:41.942814 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:42.120326 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:42.120287 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:42.360694 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:42.360648 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:42.538725 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:42.538687 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:44.166705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:44.166666 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:45.529661 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:44:45.529622 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:44:45.701679 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:45.701632 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:47.315051 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:47.315009 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:48.925329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:48.925285 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:50.528454 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:50.528408 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:51.256257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:51.256218 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:51.427656 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:51.427620 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:52.213340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:52.213300 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:53.585556 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:53.585517 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:53.760590 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:53.760552 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:53.927349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:53.927252 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:55.416386 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:55.416344 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:57.000824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:57.000757 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:58.628757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:44:58.628698 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:00.156694 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.156653 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr"] Oct 29 22:45:00.157690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157660 4952 cpu_manager.go:414] "containerMap: removing container" podUID="bc7020b1-ea9e-450d-936b-40965f599b95" containerName="maas-api" Oct 29 22:45:00.157690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157691 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7020b1-ea9e-450d-936b-40965f599b95" containerName="maas-api" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157715 4952 cpu_manager.go:414] "containerMap: removing container" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157723 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157737 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5a693545-a2b8-41c3-984c-74905ea0b217" containerName="maas-api" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157746 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a693545-a2b8-41c3-984c-74905ea0b217" containerName="maas-api" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157764 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5d367b46-cebf-42bf-93d4-a5c558ec1448" containerName="maas-api" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157771 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d367b46-cebf-42bf-93d4-a5c558ec1448" containerName="maas-api" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157788 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157794 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157803 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157810 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157826 4952 cpu_manager.go:414] "containerMap: removing container" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157832 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157840 4952 cpu_manager.go:414] "containerMap: removing container" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157847 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157859 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157867 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157876 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157883 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157897 4952 cpu_manager.go:414] "containerMap: removing container" podUID="64fa5b64-54c9-4cb9-aa39-771246749378" containerName="discovery" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157918 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64fa5b64-54c9-4cb9-aa39-771246749378" containerName="discovery" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157931 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157938 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" Oct 29 22:45:00.157942 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157951 4952 cpu_manager.go:414] "containerMap: removing container" podUID="35243f45-e93c-48e5-aa76-f9cefdb58328" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157958 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35243f45-e93c-48e5-aa76-f9cefdb58328" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157976 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.157985 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158005 4952 cpu_manager.go:414] "containerMap: removing container" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158012 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158021 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9cf35311-cf34-4be6-847c-4088a59bf188" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158029 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf35311-cf34-4be6-847c-4088a59bf188" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158042 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9f169249-a54d-45cc-a71e-52a5bd54d81d" containerName="maas-api" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158050 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f169249-a54d-45cc-a71e-52a5bd54d81d" containerName="maas-api" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158066 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158073 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158084 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158092 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158102 4952 cpu_manager.go:414] "containerMap: removing container" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158109 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158123 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158130 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158144 4952 cpu_manager.go:414] "containerMap: removing container" podUID="1deeeb21-199d-4f40-81c6-192ce3dbb82a" containerName="maas-api" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158151 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1deeeb21-199d-4f40-81c6-192ce3dbb82a" containerName="maas-api" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158327 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="52740807-cfc4-4144-b4dc-14ef28aff6d6" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158344 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a693545-a2b8-41c3-984c-74905ea0b217" containerName="maas-api" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158357 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d367b46-cebf-42bf-93d4-a5c558ec1448" containerName="maas-api" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158367 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e290a6d-f30b-4005-b567-94091995ec44" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158377 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158387 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="de8d8749-f5c9-4bbc-870a-bd919a44dec7" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158395 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="068d660a-401c-4b0a-9717-3a162259439a" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158405 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a2de66-623a-4ede-af37-c683517ca3f4" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158415 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84b5278-d243-470c-abf8-e6e8e9aa3e62" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158427 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="64fa5b64-54c9-4cb9-aa39-771246749378" containerName="discovery" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158435 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a012f50f-b410-4b94-96bb-1ea578bc090b" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158444 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="35243f45-e93c-48e5-aa76-f9cefdb58328" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158456 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e971f8a4-abc5-4235-8abb-4f26d225e35b" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158466 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158477 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf35311-cf34-4be6-847c-4088a59bf188" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158485 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f169249-a54d-45cc-a71e-52a5bd54d81d" containerName="maas-api" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158494 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e67b8901-4c7f-4f58-a38e-6da70147e552" containerName="main" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158502 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff926c55-e2dc-49b5-9e20-ca7805198914" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158512 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="79727b77-ad57-4b9d-b0fd-32a97fe5e4c1" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158522 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6911000-63f9-412a-a097-ed410b7ce727" containerName="istio-proxy" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158530 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="1deeeb21-199d-4f40-81c6-192ce3dbb82a" containerName="maas-api" Oct 29 22:45:00.159169 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.158540 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7020b1-ea9e-450d-936b-40965f599b95" containerName="maas-api" Oct 29 22:45:00.162232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.162210 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw"] Oct 29 22:45:00.162350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.162328 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" Oct 29 22:45:00.167019 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.166987 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" Oct 29 22:45:00.185027 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.184777 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk"] Oct 29 22:45:00.189322 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.189299 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj"] Oct 29 22:45:00.189462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.189432 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.194316 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.195257 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x"] Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.195813 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.200034 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.201253 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr"] Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.209128 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw"] Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.216179 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m"] Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.223828 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87dc\" (UniqueName: \"kubernetes.io/projected/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-kube-api-access-q87dc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr\" (UID: \"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.223934 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr\" (UID: \"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.223997 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw\" (UID: \"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.224144 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx2fc\" (UniqueName: \"kubernetes.io/projected/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-kube-api-access-nx2fc\") pod \"maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw\" (UID: \"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.253092 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p"] Oct 29 22:45:00.257030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.253220 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" Oct 29 22:45:00.291892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.259342 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk"] Oct 29 22:45:00.291892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.259465 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" Oct 29 22:45:00.291892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.263787 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx"] Oct 29 22:45:00.291892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.276730 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:00.291892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.281617 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x"] Oct 29 22:45:00.291892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.281740 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" Oct 29 22:45:00.291892 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.290704 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj"] Oct 29 22:45:00.296279 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.296249 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m"] Oct 29 22:45:00.299356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.299333 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5"] Oct 29 22:45:00.304488 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.303805 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p"] Oct 29 22:45:00.304488 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.303971 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" Oct 29 22:45:00.311451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.310432 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5"] Oct 29 22:45:00.327982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.322959 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx"] Oct 29 22:45:00.328194 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328132 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx2fc\" (UniqueName: \"kubernetes.io/projected/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-kube-api-access-nx2fc\") pod \"maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw\" (UID: \"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" Oct 29 22:45:00.328274 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328200 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037a603f-52cf-4222-a5be-57f8cb73dc98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk\" (UID: \"037a603f-52cf-4222-a5be-57f8cb73dc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" Oct 29 22:45:00.328274 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328233 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713cded9-f796-464a-8fed-59dfbcedb3d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p\" (UID: \"713cded9-f796-464a-8fed-59dfbcedb3d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" Oct 29 22:45:00.328421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328276 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3dfc270-04cc-4779-a455-774faa79037e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m\" (UID: \"c3dfc270-04cc-4779-a455-774faa79037e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" Oct 29 22:45:00.328421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328316 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a24bdb-4f05-4544-bd37-1896c18dffa9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x\" (UID: \"93a24bdb-4f05-4544-bd37-1896c18dffa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" Oct 29 22:45:00.328421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328365 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87dc\" (UniqueName: \"kubernetes.io/projected/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-kube-api-access-q87dc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr\" (UID: \"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" Oct 29 22:45:00.328421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328405 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4td9\" (UniqueName: \"kubernetes.io/projected/713cded9-f796-464a-8fed-59dfbcedb3d7-kube-api-access-c4td9\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p\" (UID: \"713cded9-f796-464a-8fed-59dfbcedb3d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" Oct 29 22:45:00.328639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328468 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr\" (UID: \"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" Oct 29 22:45:00.328639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328504 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkz6\" (UniqueName: \"kubernetes.io/projected/c3dfc270-04cc-4779-a455-774faa79037e-kube-api-access-2fkz6\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m\" (UID: \"c3dfc270-04cc-4779-a455-774faa79037e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" Oct 29 22:45:00.328639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328566 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw\" (UID: \"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" Oct 29 22:45:00.328639 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328611 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpk5\" (UniqueName: \"kubernetes.io/projected/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-kube-api-access-7vpk5\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj\" (UID: \"6fec9dea-1cd0-41f4-93c1-f19617b84ca1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" Oct 29 22:45:00.328827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328662 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj\" (UID: \"6fec9dea-1cd0-41f4-93c1-f19617b84ca1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" Oct 29 22:45:00.328827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328706 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvqsq\" (UniqueName: \"kubernetes.io/projected/037a603f-52cf-4222-a5be-57f8cb73dc98-kube-api-access-hvqsq\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk\" (UID: \"037a603f-52cf-4222-a5be-57f8cb73dc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" Oct 29 22:45:00.328827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.328789 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k9kf\" (UniqueName: \"kubernetes.io/projected/93a24bdb-4f05-4544-bd37-1896c18dffa9-kube-api-access-8k9kf\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x\" (UID: \"93a24bdb-4f05-4544-bd37-1896c18dffa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" Oct 29 22:45:00.330307 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.330278 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr\" (UID: \"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" Oct 29 22:45:00.331017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.330991 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw\" (UID: \"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" Oct 29 22:45:00.355642 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.355601 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87dc\" (UniqueName: \"kubernetes.io/projected/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-kube-api-access-q87dc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr\" (UID: \"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" Oct 29 22:45:00.355829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.355803 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx2fc\" (UniqueName: \"kubernetes.io/projected/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-kube-api-access-nx2fc\") pod \"maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw\" (UID: \"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" Oct 29 22:45:00.430557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.430452 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888565-jhhc5\" (UID: \"a9ae5e2e-4abe-4ac4-8a68-0773b4098020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" Oct 29 22:45:00.430557 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.430508 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k9kf\" (UniqueName: \"kubernetes.io/projected/93a24bdb-4f05-4544-bd37-1896c18dffa9-kube-api-access-8k9kf\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x\" (UID: \"93a24bdb-4f05-4544-bd37-1896c18dffa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" Oct 29 22:45:00.430793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.430614 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037a603f-52cf-4222-a5be-57f8cb73dc98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk\" (UID: \"037a603f-52cf-4222-a5be-57f8cb73dc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" Oct 29 22:45:00.430854 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.430815 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713cded9-f796-464a-8fed-59dfbcedb3d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p\" (UID: \"713cded9-f796-464a-8fed-59dfbcedb3d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" Oct 29 22:45:00.430936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.430880 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3dfc270-04cc-4779-a455-774faa79037e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m\" (UID: \"c3dfc270-04cc-4779-a455-774faa79037e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" Oct 29 22:45:00.431031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431016 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a24bdb-4f05-4544-bd37-1896c18dffa9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x\" (UID: \"93a24bdb-4f05-4544-bd37-1896c18dffa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" Oct 29 22:45:00.431114 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431078 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4td9\" (UniqueName: \"kubernetes.io/projected/713cded9-f796-464a-8fed-59dfbcedb3d7-kube-api-access-c4td9\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p\" (UID: \"713cded9-f796-464a-8fed-59dfbcedb3d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" Oct 29 22:45:00.431175 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431137 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lsbn\" (UniqueName: \"kubernetes.io/projected/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-kube-api-access-7lsbn\") pod \"maas-api-key-cleanup-29888565-jhhc5\" (UID: \"a9ae5e2e-4abe-4ac4-8a68-0773b4098020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" Oct 29 22:45:00.431236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431172 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce3d7ba-803e-45c6-a601-33268ea9223a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx\" (UID: \"cce3d7ba-803e-45c6-a601-33268ea9223a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" Oct 29 22:45:00.431236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431212 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkz6\" (UniqueName: \"kubernetes.io/projected/c3dfc270-04cc-4779-a455-774faa79037e-kube-api-access-2fkz6\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m\" (UID: \"c3dfc270-04cc-4779-a455-774faa79037e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" Oct 29 22:45:00.431350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431282 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhzk\" (UniqueName: \"kubernetes.io/projected/cce3d7ba-803e-45c6-a601-33268ea9223a-kube-api-access-xqhzk\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx\" (UID: \"cce3d7ba-803e-45c6-a601-33268ea9223a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" Oct 29 22:45:00.431350 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431328 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpk5\" (UniqueName: \"kubernetes.io/projected/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-kube-api-access-7vpk5\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj\" (UID: \"6fec9dea-1cd0-41f4-93c1-f19617b84ca1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" Oct 29 22:45:00.431468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431372 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037a603f-52cf-4222-a5be-57f8cb73dc98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk\" (UID: \"037a603f-52cf-4222-a5be-57f8cb73dc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" Oct 29 22:45:00.431468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431386 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj\" (UID: \"6fec9dea-1cd0-41f4-93c1-f19617b84ca1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" Oct 29 22:45:00.431468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431426 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvqsq\" (UniqueName: \"kubernetes.io/projected/037a603f-52cf-4222-a5be-57f8cb73dc98-kube-api-access-hvqsq\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk\" (UID: \"037a603f-52cf-4222-a5be-57f8cb73dc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" Oct 29 22:45:00.431739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431712 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3dfc270-04cc-4779-a455-774faa79037e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m\" (UID: \"c3dfc270-04cc-4779-a455-774faa79037e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" Oct 29 22:45:00.431960 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.431895 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713cded9-f796-464a-8fed-59dfbcedb3d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p\" (UID: \"713cded9-f796-464a-8fed-59dfbcedb3d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" Oct 29 22:45:00.432098 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.432084 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj\" (UID: \"6fec9dea-1cd0-41f4-93c1-f19617b84ca1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" Oct 29 22:45:00.432273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.432251 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a24bdb-4f05-4544-bd37-1896c18dffa9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x\" (UID: \"93a24bdb-4f05-4544-bd37-1896c18dffa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" Oct 29 22:45:00.457936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.457841 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k9kf\" (UniqueName: \"kubernetes.io/projected/93a24bdb-4f05-4544-bd37-1896c18dffa9-kube-api-access-8k9kf\") pod \"maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x\" (UID: \"93a24bdb-4f05-4544-bd37-1896c18dffa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" Oct 29 22:45:00.457936 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.457890 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkz6\" (UniqueName: \"kubernetes.io/projected/c3dfc270-04cc-4779-a455-774faa79037e-kube-api-access-2fkz6\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m\" (UID: \"c3dfc270-04cc-4779-a455-774faa79037e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" Oct 29 22:45:00.458231 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.457890 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpk5\" (UniqueName: \"kubernetes.io/projected/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-kube-api-access-7vpk5\") pod \"maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj\" (UID: \"6fec9dea-1cd0-41f4-93c1-f19617b84ca1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" Oct 29 22:45:00.458589 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.458558 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvqsq\" (UniqueName: \"kubernetes.io/projected/037a603f-52cf-4222-a5be-57f8cb73dc98-kube-api-access-hvqsq\") pod \"maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk\" (UID: \"037a603f-52cf-4222-a5be-57f8cb73dc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" Oct 29 22:45:00.458853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.458837 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4td9\" (UniqueName: \"kubernetes.io/projected/713cded9-f796-464a-8fed-59dfbcedb3d7-kube-api-access-c4td9\") pod \"maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p\" (UID: \"713cded9-f796-464a-8fed-59dfbcedb3d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" Oct 29 22:45:00.481387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.481344 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" Oct 29 22:45:00.497294 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.497263 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" Oct 29 22:45:00.533851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.532208 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lsbn\" (UniqueName: \"kubernetes.io/projected/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-kube-api-access-7lsbn\") pod \"maas-api-key-cleanup-29888565-jhhc5\" (UID: \"a9ae5e2e-4abe-4ac4-8a68-0773b4098020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" Oct 29 22:45:00.533851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.532268 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce3d7ba-803e-45c6-a601-33268ea9223a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx\" (UID: \"cce3d7ba-803e-45c6-a601-33268ea9223a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" Oct 29 22:45:00.533851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.532333 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhzk\" (UniqueName: \"kubernetes.io/projected/cce3d7ba-803e-45c6-a601-33268ea9223a-kube-api-access-xqhzk\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx\" (UID: \"cce3d7ba-803e-45c6-a601-33268ea9223a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" Oct 29 22:45:00.533851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.532442 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888565-jhhc5\" (UID: \"a9ae5e2e-4abe-4ac4-8a68-0773b4098020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" Oct 29 22:45:00.533851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.533357 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888565-jhhc5\" (UID: \"a9ae5e2e-4abe-4ac4-8a68-0773b4098020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" Oct 29 22:45:00.535210 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.535165 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce3d7ba-803e-45c6-a601-33268ea9223a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx\" (UID: \"cce3d7ba-803e-45c6-a601-33268ea9223a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" Oct 29 22:45:00.540485 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.540456 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" Oct 29 22:45:00.559813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.559401 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lsbn\" (UniqueName: \"kubernetes.io/projected/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-kube-api-access-7lsbn\") pod \"maas-api-key-cleanup-29888565-jhhc5\" (UID: \"a9ae5e2e-4abe-4ac4-8a68-0773b4098020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" Oct 29 22:45:00.559813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.559610 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" Oct 29 22:45:00.561529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.561470 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhzk\" (UniqueName: \"kubernetes.io/projected/cce3d7ba-803e-45c6-a601-33268ea9223a-kube-api-access-xqhzk\") pod \"maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx\" (UID: \"cce3d7ba-803e-45c6-a601-33268ea9223a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" Oct 29 22:45:00.573457 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.573246 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" Oct 29 22:45:00.598330 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.598254 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" Oct 29 22:45:00.602537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.602493 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" Oct 29 22:45:00.615389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.615148 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" Oct 29 22:45:00.645291 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.642320 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" Oct 29 22:45:00.815786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.815205 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr"] Oct 29 22:45:00.825799 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:45:00.825754 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01ec1be_c7c4_4c50_bcb5_67e4b33ab788.slice/crio-214ebb72e11f651b656f11198cd6d97bd0280540e6d670d3fc789e4dcb9e9992 WatchSource:0}: Error finding container 214ebb72e11f651b656f11198cd6d97bd0280540e6d670d3fc789e4dcb9e9992: Status 404 returned error can't find the container with id 214ebb72e11f651b656f11198cd6d97bd0280540e6d670d3fc789e4dcb9e9992 Oct 29 22:45:00.844247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.844224 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw"] Oct 29 22:45:00.915613 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.915586 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj"] Oct 29 22:45:00.940196 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:00.940120 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk"] Oct 29 22:45:01.027225 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:45:01.027112 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037a603f_52cf_4222_a5be_57f8cb73dc98.slice/crio-fb9872f398c1e373233e941e16e19f478eec9093919e56c9e2dd5b85a5685020 WatchSource:0}: Error finding container fb9872f398c1e373233e941e16e19f478eec9093919e56c9e2dd5b85a5685020: Status 404 returned error can't find the container with id fb9872f398c1e373233e941e16e19f478eec9093919e56c9e2dd5b85a5685020 Oct 29 22:45:01.039899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.039836 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5"] Oct 29 22:45:01.045617 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:45:01.045577 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ae5e2e_4abe_4ac4_8a68_0773b4098020.slice/crio-12ce90537599c7788754db9186eab46fa6e1cdae6b637279a889e450d5a0b570 WatchSource:0}: Error finding container 12ce90537599c7788754db9186eab46fa6e1cdae6b637279a889e450d5a0b570: Status 404 returned error can't find the container with id 12ce90537599c7788754db9186eab46fa6e1cdae6b637279a889e450d5a0b570 Oct 29 22:45:01.066371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.065505 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx"] Oct 29 22:45:01.071073 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:45:01.071030 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce3d7ba_803e_45c6_a601_33268ea9223a.slice/crio-5b293712fb7c6ee3d32bf43e39daed7196208b02cdc1bd1050be1a9e848a1337 WatchSource:0}: Error finding container 5b293712fb7c6ee3d32bf43e39daed7196208b02cdc1bd1050be1a9e848a1337: Status 404 returned error can't find the container with id 5b293712fb7c6ee3d32bf43e39daed7196208b02cdc1bd1050be1a9e848a1337 Oct 29 22:45:01.191074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.191007 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x"] Oct 29 22:45:01.226897 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:45:01.226852 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a24bdb_4f05_4544_bd37_1896c18dffa9.slice/crio-4cc21d1e0f0a9dd29341adcd6f9630b126d898f5da834e468dc014f3a75e5ef0 WatchSource:0}: Error finding container 4cc21d1e0f0a9dd29341adcd6f9630b126d898f5da834e468dc014f3a75e5ef0: Status 404 returned error can't find the container with id 4cc21d1e0f0a9dd29341adcd6f9630b126d898f5da834e468dc014f3a75e5ef0 Oct 29 22:45:01.245537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.245503 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p"] Oct 29 22:45:01.251950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.251919 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m"] Oct 29 22:45:01.259411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.259381 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" event={"ID":"cce3d7ba-803e-45c6-a601-33268ea9223a","Type":"ContainerStarted","Data":"5b293712fb7c6ee3d32bf43e39daed7196208b02cdc1bd1050be1a9e848a1337"} Oct 29 22:45:01.261154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.261120 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" event={"ID":"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788","Type":"ContainerStarted","Data":"c0267c96807d476db565994fa8244410df45ad20b5959da53320c61db6c9a1c9"} Oct 29 22:45:01.261273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.261168 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" event={"ID":"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788","Type":"ContainerStarted","Data":"214ebb72e11f651b656f11198cd6d97bd0280540e6d670d3fc789e4dcb9e9992"} Oct 29 22:45:01.262451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.262420 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" event={"ID":"93a24bdb-4f05-4544-bd37-1896c18dffa9","Type":"ContainerStarted","Data":"4cc21d1e0f0a9dd29341adcd6f9630b126d898f5da834e468dc014f3a75e5ef0"} Oct 29 22:45:01.263686 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.263655 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" event={"ID":"6fec9dea-1cd0-41f4-93c1-f19617b84ca1","Type":"ContainerStarted","Data":"86642b8e87e2ca67683a5c4c272690f44e92f31cb2a17ad059cbc2d78916d204"} Oct 29 22:45:01.264817 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.264796 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" event={"ID":"037a603f-52cf-4222-a5be-57f8cb73dc98","Type":"ContainerStarted","Data":"fb9872f398c1e373233e941e16e19f478eec9093919e56c9e2dd5b85a5685020"} Oct 29 22:45:01.265952 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.265893 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" event={"ID":"a9ae5e2e-4abe-4ac4-8a68-0773b4098020","Type":"ContainerStarted","Data":"12ce90537599c7788754db9186eab46fa6e1cdae6b637279a889e450d5a0b570"} Oct 29 22:45:01.267396 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.267377 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" event={"ID":"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c","Type":"ContainerStarted","Data":"12340e11182e472889d18fe56b8c4b6b64da666016a8776b3bcaabb51a6c9777"} Oct 29 22:45:01.267470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.267410 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" event={"ID":"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c","Type":"ContainerStarted","Data":"c002aa5d2de7ecf9634c6988eb46a4f406a1b1ecd1479122a6733f07b223fcac"} Oct 29 22:45:01.306596 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.306538 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" podStartSLOduration=1.306523667 podStartE2EDuration="1.306523667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:01.303265099 +0000 UTC m=+3766.777853516" watchObservedRunningTime="2026-10-29 22:45:01.306523667 +0000 UTC m=+3766.781112079" Oct 29 22:45:01.349398 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.349340 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" podStartSLOduration=1.349322468 podStartE2EDuration="1.349322468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:01.347428924 +0000 UTC m=+3766.822017343" watchObservedRunningTime="2026-10-29 22:45:01.349322468 +0000 UTC m=+3766.823910888" Oct 29 22:45:01.941762 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:01.941560 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:02.278333 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.278296 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" event={"ID":"6fec9dea-1cd0-41f4-93c1-f19617b84ca1","Type":"ContainerStarted","Data":"7105923c3ec01c62d1e49128c2b71fcc566362dceb8e7b8964e066fff3e44995"} Oct 29 22:45:02.279950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.279876 4952 generic.go:356] "Generic (PLEG): container finished" podID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerID="2aa5e2121800f619144b337dabcd5cf851bc6a0e2d90d3a04d91c5f4ee1a9f4a" exitCode=60 Oct 29 22:45:02.279950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.279936 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" event={"ID":"037a603f-52cf-4222-a5be-57f8cb73dc98","Type":"ContainerDied","Data":"2aa5e2121800f619144b337dabcd5cf851bc6a0e2d90d3a04d91c5f4ee1a9f4a"} Oct 29 22:45:02.280204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.280186 4952 scope.go:122] "RemoveContainer" containerID="2aa5e2121800f619144b337dabcd5cf851bc6a0e2d90d3a04d91c5f4ee1a9f4a" Oct 29 22:45:02.281666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.281632 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" event={"ID":"a9ae5e2e-4abe-4ac4-8a68-0773b4098020","Type":"ContainerStarted","Data":"67783e2178eb1918d2b81683afbf4117ee8de4bff10a67ea1b9057c7f5f110e0"} Oct 29 22:45:02.283360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.283341 4952 generic.go:356] "Generic (PLEG): container finished" podID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerID="12340e11182e472889d18fe56b8c4b6b64da666016a8776b3bcaabb51a6c9777" exitCode=60 Oct 29 22:45:02.283462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.283433 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" event={"ID":"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c","Type":"ContainerDied","Data":"12340e11182e472889d18fe56b8c4b6b64da666016a8776b3bcaabb51a6c9777"} Oct 29 22:45:02.283732 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.283714 4952 scope.go:122] "RemoveContainer" containerID="12340e11182e472889d18fe56b8c4b6b64da666016a8776b3bcaabb51a6c9777" Oct 29 22:45:02.285321 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.285300 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" event={"ID":"cce3d7ba-803e-45c6-a601-33268ea9223a","Type":"ContainerStarted","Data":"aca6d88f0b44fb9b22a064fa2436bdd5b0ddf85602007d987405f9a0300544d2"} Oct 29 22:45:02.286967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.286949 4952 generic.go:356] "Generic (PLEG): container finished" podID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerID="c0267c96807d476db565994fa8244410df45ad20b5959da53320c61db6c9a1c9" exitCode=60 Oct 29 22:45:02.287081 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.287032 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" event={"ID":"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788","Type":"ContainerDied","Data":"c0267c96807d476db565994fa8244410df45ad20b5959da53320c61db6c9a1c9"} Oct 29 22:45:02.287254 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.287234 4952 scope.go:122] "RemoveContainer" containerID="c0267c96807d476db565994fa8244410df45ad20b5959da53320c61db6c9a1c9" Oct 29 22:45:02.288697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.288674 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" event={"ID":"713cded9-f796-464a-8fed-59dfbcedb3d7","Type":"ContainerStarted","Data":"ac8b6f3424184df23ebfaa0c2a01e25edb01e2272608ba4c3c079c206b44c211"} Oct 29 22:45:02.288789 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.288721 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" event={"ID":"713cded9-f796-464a-8fed-59dfbcedb3d7","Type":"ContainerStarted","Data":"fde77886f7ac125027820999c4a987cb3424ac5a333d4eca79b27cd38ff976e5"} Oct 29 22:45:02.290407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.290379 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" event={"ID":"93a24bdb-4f05-4544-bd37-1896c18dffa9","Type":"ContainerStarted","Data":"52d03b4e30e02131e0950af37cad903f105368b3b4425e761871d2b03be2fb74"} Oct 29 22:45:02.292149 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.292120 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" event={"ID":"c3dfc270-04cc-4779-a455-774faa79037e","Type":"ContainerStarted","Data":"0be88a5cdae6ef95587a733c7d05c3aa77d7224878dbd58d1eb35229f6f4b3b4"} Oct 29 22:45:02.292249 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.292170 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" event={"ID":"c3dfc270-04cc-4779-a455-774faa79037e","Type":"ContainerStarted","Data":"1fb6f488e2621e58d44d330a7dd3790e9034c3afbbc13e2efde9dde4a1f864c7"} Oct 29 22:45:02.382621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.382563 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" podStartSLOduration=2.382548562 podStartE2EDuration="2.382548562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.334968255 +0000 UTC m=+3767.809556674" watchObservedRunningTime="2026-10-29 22:45:02.382548562 +0000 UTC m=+3767.857136996" Oct 29 22:45:02.428238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.427428 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" podStartSLOduration=2.427412831 podStartE2EDuration="2.427412831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.423950759 +0000 UTC m=+3767.898539180" watchObservedRunningTime="2026-10-29 22:45:02.427412831 +0000 UTC m=+3767.902001248" Oct 29 22:45:02.514387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.514337 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" podStartSLOduration=2.514322547 podStartE2EDuration="2.514322547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.51174121 +0000 UTC m=+3767.986329628" watchObservedRunningTime="2026-10-29 22:45:02.514322547 +0000 UTC m=+3767.988911032" Oct 29 22:45:02.602219 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.602151 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" podStartSLOduration=2.602135981 podStartE2EDuration="2.602135981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.599520283 +0000 UTC m=+3768.074108700" watchObservedRunningTime="2026-10-29 22:45:02.602135981 +0000 UTC m=+3768.076724399" Oct 29 22:45:02.644208 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.644120 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" podStartSLOduration=2.644102056 podStartE2EDuration="2.644102056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.642800815 +0000 UTC m=+3768.117389236" watchObservedRunningTime="2026-10-29 22:45:02.644102056 +0000 UTC m=+3768.118690478" Oct 29 22:45:02.690378 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:02.690326 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" podStartSLOduration=2.690312181 podStartE2EDuration="2.690312181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.689597064 +0000 UTC m=+3768.164185493" watchObservedRunningTime="2026-10-29 22:45:02.690312181 +0000 UTC m=+3768.164900598" Oct 29 22:45:03.220862 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.220820 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:45:03.315968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.314916 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" event={"ID":"037a603f-52cf-4222-a5be-57f8cb73dc98","Type":"ContainerStarted","Data":"a796f6f1375c4a664f45d66ecb1174f4c1c8902d04ca74ec6d9eacd0ddbe01d7"} Oct 29 22:45:03.362915 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.362761 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm"] Oct 29 22:45:03.365970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.365361 4952 generic.go:356] "Generic (PLEG): container finished" podID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerID="67783e2178eb1918d2b81683afbf4117ee8de4bff10a67ea1b9057c7f5f110e0" exitCode=60 Oct 29 22:45:03.365970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.365453 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" event={"ID":"a9ae5e2e-4abe-4ac4-8a68-0773b4098020","Type":"ContainerDied","Data":"67783e2178eb1918d2b81683afbf4117ee8de4bff10a67ea1b9057c7f5f110e0"} Oct 29 22:45:03.365970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.365858 4952 scope.go:122] "RemoveContainer" containerID="67783e2178eb1918d2b81683afbf4117ee8de4bff10a67ea1b9057c7f5f110e0" Oct 29 22:45:03.376951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.370405 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" event={"ID":"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c","Type":"ContainerStarted","Data":"3460905d0550f14b451bc0989bd0fbc2dde9bade25e83fd2dc22433aaa707e77"} Oct 29 22:45:03.387494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.387470 4952 generic.go:356] "Generic (PLEG): container finished" podID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerID="aca6d88f0b44fb9b22a064fa2436bdd5b0ddf85602007d987405f9a0300544d2" exitCode=60 Oct 29 22:45:03.387741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.387723 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" event={"ID":"cce3d7ba-803e-45c6-a601-33268ea9223a","Type":"ContainerDied","Data":"aca6d88f0b44fb9b22a064fa2436bdd5b0ddf85602007d987405f9a0300544d2"} Oct 29 22:45:03.388153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.388138 4952 scope.go:122] "RemoveContainer" containerID="aca6d88f0b44fb9b22a064fa2436bdd5b0ddf85602007d987405f9a0300544d2" Oct 29 22:45:03.397975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.393148 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" podStartSLOduration=3.393130446 podStartE2EDuration="3.393130446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.389629396 +0000 UTC m=+3768.864217815" watchObservedRunningTime="2026-10-29 22:45:03.393130446 +0000 UTC m=+3768.867718866" Oct 29 22:45:03.410003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.405766 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" event={"ID":"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788","Type":"ContainerStarted","Data":"8e584c9abdd19fc8c2cd12387fa757aa3c09573751aefb6cec944c5bc51b7a2c"} Oct 29 22:45:03.413241 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.413207 4952 generic.go:356] "Generic (PLEG): container finished" podID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerID="ac8b6f3424184df23ebfaa0c2a01e25edb01e2272608ba4c3c079c206b44c211" exitCode=60 Oct 29 22:45:03.413475 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.413317 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" event={"ID":"713cded9-f796-464a-8fed-59dfbcedb3d7","Type":"ContainerDied","Data":"ac8b6f3424184df23ebfaa0c2a01e25edb01e2272608ba4c3c079c206b44c211"} Oct 29 22:45:03.413615 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.413592 4952 scope.go:122] "RemoveContainer" containerID="ac8b6f3424184df23ebfaa0c2a01e25edb01e2272608ba4c3c079c206b44c211" Oct 29 22:45:03.418633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.416976 4952 generic.go:356] "Generic (PLEG): container finished" podID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerID="52d03b4e30e02131e0950af37cad903f105368b3b4425e761871d2b03be2fb74" exitCode=60 Oct 29 22:45:03.418633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.417551 4952 scope.go:122] "RemoveContainer" containerID="52d03b4e30e02131e0950af37cad903f105368b3b4425e761871d2b03be2fb74" Oct 29 22:45:03.419238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.418985 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" event={"ID":"93a24bdb-4f05-4544-bd37-1896c18dffa9","Type":"ContainerDied","Data":"52d03b4e30e02131e0950af37cad903f105368b3b4425e761871d2b03be2fb74"} Oct 29 22:45:03.433536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.433504 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:03.499362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.498626 4952 generic.go:356] "Generic (PLEG): container finished" podID="c3dfc270-04cc-4779-a455-774faa79037e" containerID="0be88a5cdae6ef95587a733c7d05c3aa77d7224878dbd58d1eb35229f6f4b3b4" exitCode=60 Oct 29 22:45:03.499362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.498792 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" event={"ID":"c3dfc270-04cc-4779-a455-774faa79037e","Type":"ContainerDied","Data":"0be88a5cdae6ef95587a733c7d05c3aa77d7224878dbd58d1eb35229f6f4b3b4"} Oct 29 22:45:03.504600 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.503987 4952 scope.go:122] "RemoveContainer" containerID="0be88a5cdae6ef95587a733c7d05c3aa77d7224878dbd58d1eb35229f6f4b3b4" Oct 29 22:45:03.519312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.519277 4952 generic.go:356] "Generic (PLEG): container finished" podID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerID="7105923c3ec01c62d1e49128c2b71fcc566362dceb8e7b8964e066fff3e44995" exitCode=60 Oct 29 22:45:03.519427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.519340 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" event={"ID":"6fec9dea-1cd0-41f4-93c1-f19617b84ca1","Type":"ContainerDied","Data":"7105923c3ec01c62d1e49128c2b71fcc566362dceb8e7b8964e066fff3e44995"} Oct 29 22:45:03.519806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.519783 4952 scope.go:122] "RemoveContainer" containerID="7105923c3ec01c62d1e49128c2b71fcc566362dceb8e7b8964e066fff3e44995" Oct 29 22:45:03.761481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:03.761441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:04.541879 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.541546 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" event={"ID":"6fec9dea-1cd0-41f4-93c1-f19617b84ca1","Type":"ContainerStarted","Data":"16961e4cfb4d1793f52e1b00c45c62991c672e15f32108553c652e7b12da6e14"} Oct 29 22:45:04.551829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.551795 4952 generic.go:356] "Generic (PLEG): container finished" podID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerID="a796f6f1375c4a664f45d66ecb1174f4c1c8902d04ca74ec6d9eacd0ddbe01d7" exitCode=60 Oct 29 22:45:04.552036 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.551945 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" event={"ID":"037a603f-52cf-4222-a5be-57f8cb73dc98","Type":"ContainerDied","Data":"a796f6f1375c4a664f45d66ecb1174f4c1c8902d04ca74ec6d9eacd0ddbe01d7"} Oct 29 22:45:04.552140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.552064 4952 scope.go:122] "RemoveContainer" containerID="2aa5e2121800f619144b337dabcd5cf851bc6a0e2d90d3a04d91c5f4ee1a9f4a" Oct 29 22:45:04.552293 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.552268 4952 scope.go:122] "RemoveContainer" containerID="a796f6f1375c4a664f45d66ecb1174f4c1c8902d04ca74ec6d9eacd0ddbe01d7" Oct 29 22:45:04.552960 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:04.552929 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk_odh-ai-gateway-infra(037a603f-52cf-4222-a5be-57f8cb73dc98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" Oct 29 22:45:04.555159 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.555137 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" event={"ID":"a9ae5e2e-4abe-4ac4-8a68-0773b4098020","Type":"ContainerStarted","Data":"97029fec14c62832191a51e7432e5c8eff0e11fd5da7d3158c6f439d3c894dd9"} Oct 29 22:45:04.560787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.555689 4952 scope.go:122] "RemoveContainer" containerID="97029fec14c62832191a51e7432e5c8eff0e11fd5da7d3158c6f439d3c894dd9" Oct 29 22:45:04.560787 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:04.556043 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888565-jhhc5_odh-ai-gateway-infra(a9ae5e2e-4abe-4ac4-8a68-0773b4098020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" Oct 29 22:45:04.560787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.557486 4952 generic.go:356] "Generic (PLEG): container finished" podID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerID="3460905d0550f14b451bc0989bd0fbc2dde9bade25e83fd2dc22433aaa707e77" exitCode=60 Oct 29 22:45:04.560787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.557598 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" event={"ID":"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c","Type":"ContainerDied","Data":"3460905d0550f14b451bc0989bd0fbc2dde9bade25e83fd2dc22433aaa707e77"} Oct 29 22:45:04.560787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.558232 4952 scope.go:122] "RemoveContainer" containerID="3460905d0550f14b451bc0989bd0fbc2dde9bade25e83fd2dc22433aaa707e77" Oct 29 22:45:04.560787 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:04.558497 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw_odh-ai-gateway-infra(7fc63e06-4fc4-4f20-ad12-1e3419d39a1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" Oct 29 22:45:04.562509 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.562476 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" event={"ID":"cce3d7ba-803e-45c6-a601-33268ea9223a","Type":"ContainerStarted","Data":"8e30c643597f8b87a57aca375d98b34c4e03f143c0d815b528e4b98d055e1955"} Oct 29 22:45:04.565096 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.565068 4952 generic.go:356] "Generic (PLEG): container finished" podID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerID="8e584c9abdd19fc8c2cd12387fa757aa3c09573751aefb6cec944c5bc51b7a2c" exitCode=60 Oct 29 22:45:04.565216 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.565184 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" event={"ID":"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788","Type":"ContainerDied","Data":"8e584c9abdd19fc8c2cd12387fa757aa3c09573751aefb6cec944c5bc51b7a2c"} Oct 29 22:45:04.565540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.565518 4952 scope.go:122] "RemoveContainer" containerID="8e584c9abdd19fc8c2cd12387fa757aa3c09573751aefb6cec944c5bc51b7a2c" Oct 29 22:45:04.565822 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:04.565795 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr_odh-ai-gateway-infra(b01ec1be-c7c4-4c50-bcb5-67e4b33ab788)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" Oct 29 22:45:04.569145 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.569117 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" event={"ID":"713cded9-f796-464a-8fed-59dfbcedb3d7","Type":"ContainerStarted","Data":"d7f0dc8da67b5b72b3e9e8d593e6ea4042a8fa3cea3932545252aa028ab96750"} Oct 29 22:45:04.572506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.572435 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" event={"ID":"93a24bdb-4f05-4544-bd37-1896c18dffa9","Type":"ContainerStarted","Data":"70e548bc3538ec09387a4536916b22c471d3111a01b74e54a22be9b37f698664"} Oct 29 22:45:04.575248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.575222 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" event={"ID":"c3dfc270-04cc-4779-a455-774faa79037e","Type":"ContainerStarted","Data":"6e9a0820c3aacd2cc6809423142f9f7822debc2602fdf9a7f73528b70600936d"} Oct 29 22:45:04.585447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.585404 4952 scope.go:122] "RemoveContainer" containerID="12340e11182e472889d18fe56b8c4b6b64da666016a8776b3bcaabb51a6c9777" Oct 29 22:45:04.600407 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:04.600382 4952 scope.go:122] "RemoveContainer" containerID="c0267c96807d476db565994fa8244410df45ad20b5959da53320c61db6c9a1c9" Oct 29 22:45:05.244655 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.244610 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m"] Oct 29 22:45:05.317955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.317898 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:05.672693 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.672430 4952 generic.go:356] "Generic (PLEG): container finished" podID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerID="70e548bc3538ec09387a4536916b22c471d3111a01b74e54a22be9b37f698664" exitCode=60 Oct 29 22:45:05.674079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.672765 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" event={"ID":"93a24bdb-4f05-4544-bd37-1896c18dffa9","Type":"ContainerDied","Data":"70e548bc3538ec09387a4536916b22c471d3111a01b74e54a22be9b37f698664"} Oct 29 22:45:05.674079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.673334 4952 scope.go:122] "RemoveContainer" containerID="70e548bc3538ec09387a4536916b22c471d3111a01b74e54a22be9b37f698664" Oct 29 22:45:05.675012 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:05.674632 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x_odh-ai-gateway-infra(93a24bdb-4f05-4544-bd37-1896c18dffa9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" Oct 29 22:45:05.675476 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.675456 4952 scope.go:122] "RemoveContainer" containerID="52d03b4e30e02131e0950af37cad903f105368b3b4425e761871d2b03be2fb74" Oct 29 22:45:05.679768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.679744 4952 generic.go:356] "Generic (PLEG): container finished" podID="c3dfc270-04cc-4779-a455-774faa79037e" containerID="6e9a0820c3aacd2cc6809423142f9f7822debc2602fdf9a7f73528b70600936d" exitCode=60 Oct 29 22:45:05.680226 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.679945 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" event={"ID":"c3dfc270-04cc-4779-a455-774faa79037e","Type":"ContainerDied","Data":"6e9a0820c3aacd2cc6809423142f9f7822debc2602fdf9a7f73528b70600936d"} Oct 29 22:45:05.680336 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.680064 4952 scope.go:122] "RemoveContainer" containerID="6e9a0820c3aacd2cc6809423142f9f7822debc2602fdf9a7f73528b70600936d" Oct 29 22:45:05.680886 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:05.680864 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m_odh-ai-gateway-infra(c3dfc270-04cc-4779-a455-774faa79037e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" podUID="c3dfc270-04cc-4779-a455-774faa79037e" Oct 29 22:45:05.682933 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.682864 4952 generic.go:356] "Generic (PLEG): container finished" podID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerID="16961e4cfb4d1793f52e1b00c45c62991c672e15f32108553c652e7b12da6e14" exitCode=60 Oct 29 22:45:05.683191 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.683166 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" event={"ID":"6fec9dea-1cd0-41f4-93c1-f19617b84ca1","Type":"ContainerDied","Data":"16961e4cfb4d1793f52e1b00c45c62991c672e15f32108553c652e7b12da6e14"} Oct 29 22:45:05.683499 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.683479 4952 scope.go:122] "RemoveContainer" containerID="16961e4cfb4d1793f52e1b00c45c62991c672e15f32108553c652e7b12da6e14" Oct 29 22:45:05.683772 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:05.683753 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj_odh-ai-gateway-infra(6fec9dea-1cd0-41f4-93c1-f19617b84ca1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" Oct 29 22:45:05.686481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.686459 4952 scope.go:122] "RemoveContainer" containerID="a796f6f1375c4a664f45d66ecb1174f4c1c8902d04ca74ec6d9eacd0ddbe01d7" Oct 29 22:45:05.687013 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:05.686887 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk_odh-ai-gateway-infra(037a603f-52cf-4222-a5be-57f8cb73dc98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" Oct 29 22:45:05.690533 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.690501 4952 generic.go:356] "Generic (PLEG): container finished" podID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerID="97029fec14c62832191a51e7432e5c8eff0e11fd5da7d3158c6f439d3c894dd9" exitCode=60 Oct 29 22:45:05.690834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.690806 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" event={"ID":"a9ae5e2e-4abe-4ac4-8a68-0773b4098020","Type":"ContainerDied","Data":"97029fec14c62832191a51e7432e5c8eff0e11fd5da7d3158c6f439d3c894dd9"} Oct 29 22:45:05.695480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.695420 4952 scope.go:122] "RemoveContainer" containerID="0be88a5cdae6ef95587a733c7d05c3aa77d7224878dbd58d1eb35229f6f4b3b4" Oct 29 22:45:05.695822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.695785 4952 scope.go:122] "RemoveContainer" containerID="3460905d0550f14b451bc0989bd0fbc2dde9bade25e83fd2dc22433aaa707e77" Oct 29 22:45:05.696147 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:05.696122 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw_odh-ai-gateway-infra(7fc63e06-4fc4-4f20-ad12-1e3419d39a1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" Oct 29 22:45:05.696898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.696822 4952 generic.go:356] "Generic (PLEG): container finished" podID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerID="8e30c643597f8b87a57aca375d98b34c4e03f143c0d815b528e4b98d055e1955" exitCode=60 Oct 29 22:45:05.696898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.696845 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" event={"ID":"cce3d7ba-803e-45c6-a601-33268ea9223a","Type":"ContainerDied","Data":"8e30c643597f8b87a57aca375d98b34c4e03f143c0d815b528e4b98d055e1955"} Oct 29 22:45:05.697610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.697577 4952 scope.go:122] "RemoveContainer" containerID="97029fec14c62832191a51e7432e5c8eff0e11fd5da7d3158c6f439d3c894dd9" Oct 29 22:45:05.697887 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:05.697865 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888565-jhhc5_odh-ai-gateway-infra(a9ae5e2e-4abe-4ac4-8a68-0773b4098020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" Oct 29 22:45:05.698039 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.698024 4952 scope.go:122] "RemoveContainer" containerID="8e30c643597f8b87a57aca375d98b34c4e03f143c0d815b528e4b98d055e1955" Oct 29 22:45:05.698269 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:05.698250 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx_odh-ai-gateway-infra(cce3d7ba-803e-45c6-a601-33268ea9223a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" Oct 29 22:45:05.699362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.699345 4952 scope.go:122] "RemoveContainer" containerID="8e584c9abdd19fc8c2cd12387fa757aa3c09573751aefb6cec944c5bc51b7a2c" Oct 29 22:45:05.699581 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:05.699561 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr_odh-ai-gateway-infra(b01ec1be-c7c4-4c50-bcb5-67e4b33ab788)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" Oct 29 22:45:05.701413 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.701386 4952 generic.go:356] "Generic (PLEG): container finished" podID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerID="d7f0dc8da67b5b72b3e9e8d593e6ea4042a8fa3cea3932545252aa028ab96750" exitCode=60 Oct 29 22:45:05.701523 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.701432 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" event={"ID":"713cded9-f796-464a-8fed-59dfbcedb3d7","Type":"ContainerDied","Data":"d7f0dc8da67b5b72b3e9e8d593e6ea4042a8fa3cea3932545252aa028ab96750"} Oct 29 22:45:05.701685 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.701664 4952 scope.go:122] "RemoveContainer" containerID="d7f0dc8da67b5b72b3e9e8d593e6ea4042a8fa3cea3932545252aa028ab96750" Oct 29 22:45:05.701883 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:05.701857 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p_odh-ai-gateway-infra(713cded9-f796-464a-8fed-59dfbcedb3d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" Oct 29 22:45:05.712228 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.712207 4952 scope.go:122] "RemoveContainer" containerID="7105923c3ec01c62d1e49128c2b71fcc566362dceb8e7b8964e066fff3e44995" Oct 29 22:45:05.728812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.728769 4952 scope.go:122] "RemoveContainer" containerID="67783e2178eb1918d2b81683afbf4117ee8de4bff10a67ea1b9057c7f5f110e0" Oct 29 22:45:05.743135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.743109 4952 scope.go:122] "RemoveContainer" containerID="aca6d88f0b44fb9b22a064fa2436bdd5b0ddf85602007d987405f9a0300544d2" Oct 29 22:45:05.757750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.757725 4952 scope.go:122] "RemoveContainer" containerID="ac8b6f3424184df23ebfaa0c2a01e25edb01e2272608ba4c3c079c206b44c211" Oct 29 22:45:05.880539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:05.880498 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:06.052385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:06.052286 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:06.532838 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:06.532802 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:06.713013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:06.712988 4952 scope.go:122] "RemoveContainer" containerID="16961e4cfb4d1793f52e1b00c45c62991c672e15f32108553c652e7b12da6e14" Oct 29 22:45:06.713399 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:06.713331 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj_odh-ai-gateway-infra(6fec9dea-1cd0-41f4-93c1-f19617b84ca1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" Oct 29 22:45:06.715015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:06.714995 4952 scope.go:122] "RemoveContainer" containerID="97029fec14c62832191a51e7432e5c8eff0e11fd5da7d3158c6f439d3c894dd9" Oct 29 22:45:06.715506 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:06.715481 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888565-jhhc5_odh-ai-gateway-infra(a9ae5e2e-4abe-4ac4-8a68-0773b4098020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" Oct 29 22:45:06.716751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:06.716731 4952 scope.go:122] "RemoveContainer" containerID="8e30c643597f8b87a57aca375d98b34c4e03f143c0d815b528e4b98d055e1955" Oct 29 22:45:06.717015 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:06.716989 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx_odh-ai-gateway-infra(cce3d7ba-803e-45c6-a601-33268ea9223a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" Oct 29 22:45:06.719219 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:06.718888 4952 scope.go:122] "RemoveContainer" containerID="d7f0dc8da67b5b72b3e9e8d593e6ea4042a8fa3cea3932545252aa028ab96750" Oct 29 22:45:06.719219 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:06.719192 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p_odh-ai-gateway-infra(713cded9-f796-464a-8fed-59dfbcedb3d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" Oct 29 22:45:06.720824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:06.720803 4952 scope.go:122] "RemoveContainer" containerID="70e548bc3538ec09387a4536916b22c471d3111a01b74e54a22be9b37f698664" Oct 29 22:45:06.721050 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:06.721031 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x_odh-ai-gateway-infra(93a24bdb-4f05-4544-bd37-1896c18dffa9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" Oct 29 22:45:06.877574 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:06.877541 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" Oct 29 22:45:07.053165 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.053126 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3dfc270-04cc-4779-a455-774faa79037e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3dfc270-04cc-4779-a455-774faa79037e-maas-api-service-ca\") pod \"c3dfc270-04cc-4779-a455-774faa79037e\" (UID: \"c3dfc270-04cc-4779-a455-774faa79037e\") " Oct 29 22:45:07.053337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.053209 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3dfc270-04cc-4779-a455-774faa79037e-kube-api-access-2fkz6\" (UniqueName: \"kubernetes.io/projected/c3dfc270-04cc-4779-a455-774faa79037e-kube-api-access-2fkz6\") pod \"c3dfc270-04cc-4779-a455-774faa79037e\" (UID: \"c3dfc270-04cc-4779-a455-774faa79037e\") " Oct 29 22:45:07.053575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.053548 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3dfc270-04cc-4779-a455-774faa79037e-maas-api-service-ca" pod "c3dfc270-04cc-4779-a455-774faa79037e" (UID: "c3dfc270-04cc-4779-a455-774faa79037e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:07.053791 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.053768 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3dfc270-04cc-4779-a455-774faa79037e-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:07.055431 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.055399 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3dfc270-04cc-4779-a455-774faa79037e-kube-api-access-2fkz6" pod "c3dfc270-04cc-4779-a455-774faa79037e" (UID: "c3dfc270-04cc-4779-a455-774faa79037e"). InnerVolumeSpecName "kube-api-access-2fkz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:07.155511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.155416 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fkz6\" (UniqueName: \"kubernetes.io/projected/c3dfc270-04cc-4779-a455-774faa79037e-kube-api-access-2fkz6\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:07.672089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.672049 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:07.730236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.730199 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" Oct 29 22:45:07.730707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.730200 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m" event={"ID":"c3dfc270-04cc-4779-a455-774faa79037e","Type":"ContainerDied","Data":"1fb6f488e2621e58d44d330a7dd3790e9034c3afbbc13e2efde9dde4a1f864c7"} Oct 29 22:45:07.730707 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.730340 4952 scope.go:122] "RemoveContainer" containerID="6e9a0820c3aacd2cc6809423142f9f7822debc2602fdf9a7f73528b70600936d" Oct 29 22:45:07.789182 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.789143 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m"] Oct 29 22:45:07.801961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:07.801924 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d8760-29888565-wf45m"] Oct 29 22:45:08.646132 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:08.646095 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3dfc270-04cc-4779-a455-774faa79037e" path="/var/lib/kubelet/pods/c3dfc270-04cc-4779-a455-774faa79037e/volumes" Oct 29 22:45:09.341640 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:09.341596 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:10.918768 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:10.918727 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:12.502490 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:12.502449 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:13.234957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:13.234881 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:13.483434 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:13.483400 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:13.734042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:13.733980 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:14.042411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.042303 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:14.190000 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.189955 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg"] Oct 29 22:45:14.190961 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.190932 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c3dfc270-04cc-4779-a455-774faa79037e" containerName="cleanup" Oct 29 22:45:14.191044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.190962 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3dfc270-04cc-4779-a455-774faa79037e" containerName="cleanup" Oct 29 22:45:14.191044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.191001 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c3dfc270-04cc-4779-a455-774faa79037e" containerName="cleanup" Oct 29 22:45:14.191044 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.191009 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3dfc270-04cc-4779-a455-774faa79037e" containerName="cleanup" Oct 29 22:45:14.191214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.191200 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3dfc270-04cc-4779-a455-774faa79037e" containerName="cleanup" Oct 29 22:45:14.191253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.191229 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3dfc270-04cc-4779-a455-774faa79037e" containerName="cleanup" Oct 29 22:45:14.196819 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.196796 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.216441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.216407 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg"] Oct 29 22:45:14.222596 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.222563 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:14.222835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.222807 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:14.254243 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.254198 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.254487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.254404 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c44p\" (UniqueName: \"kubernetes.io/projected/95346df9-7c90-4c49-817d-aac3d90ef04f-kube-api-access-6c44p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.254487 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.254437 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.355462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.355410 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c44p\" (UniqueName: \"kubernetes.io/projected/95346df9-7c90-4c49-817d-aac3d90ef04f-kube-api-access-6c44p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.355462 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.355459 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.355787 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.355543 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.356419 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.356387 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.363385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.363356 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.382002 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.381967 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c44p\" (UniqueName: \"kubernetes.io/projected/95346df9-7c90-4c49-817d-aac3d90ef04f-kube-api-access-6c44p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.514328 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.514272 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:14.831155 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:14.831118 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg"] Oct 29 22:45:14.832432 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:45:14.832402 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95346df9_7c90_4c49_817d_aac3d90ef04f.slice/crio-4f374f144c62e4b0d47511bfa2450d1083f90848dd040646b9b7b88dbb8cae61 WatchSource:0}: Error finding container 4f374f144c62e4b0d47511bfa2450d1083f90848dd040646b9b7b88dbb8cae61: Status 404 returned error can't find the container with id 4f374f144c62e4b0d47511bfa2450d1083f90848dd040646b9b7b88dbb8cae61 Oct 29 22:45:15.529224 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:15.529126 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:45:15.637340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:15.637300 4952 scope.go:122] "RemoveContainer" containerID="97029fec14c62832191a51e7432e5c8eff0e11fd5da7d3158c6f439d3c894dd9" Oct 29 22:45:15.827834 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:15.827789 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" event={"ID":"95346df9-7c90-4c49-817d-aac3d90ef04f","Type":"ContainerStarted","Data":"0fed92b19edd1c90c5abac72b36ac62e004c42640858d2ebe4f2fdbfd231f4ea"} Oct 29 22:45:15.828100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:15.827854 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" event={"ID":"95346df9-7c90-4c49-817d-aac3d90ef04f","Type":"ContainerStarted","Data":"4f374f144c62e4b0d47511bfa2450d1083f90848dd040646b9b7b88dbb8cae61"} Oct 29 22:45:15.873535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:15.873497 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:15.880459 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:15.880397 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" podStartSLOduration=1.880376548 podStartE2EDuration="1.880376548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:15.877480226 +0000 UTC m=+3781.352068644" watchObservedRunningTime="2026-10-29 22:45:15.880376548 +0000 UTC m=+3781.354964970" Oct 29 22:45:16.638126 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:16.637885 4952 scope.go:122] "RemoveContainer" containerID="8e30c643597f8b87a57aca375d98b34c4e03f143c0d815b528e4b98d055e1955" Oct 29 22:45:16.638801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:16.638488 4952 scope.go:122] "RemoveContainer" containerID="70e548bc3538ec09387a4536916b22c471d3111a01b74e54a22be9b37f698664" Oct 29 22:45:16.650351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:16.647644 4952 scope.go:122] "RemoveContainer" containerID="3460905d0550f14b451bc0989bd0fbc2dde9bade25e83fd2dc22433aaa707e77" Oct 29 22:45:16.860982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:16.859954 4952 generic.go:356] "Generic (PLEG): container finished" podID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerID="a159c706f31762a2c36dc0f646579da837818fdc448e4144d9f02417232173ca" exitCode=60 Oct 29 22:45:16.860982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:16.860084 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" event={"ID":"a9ae5e2e-4abe-4ac4-8a68-0773b4098020","Type":"ContainerDied","Data":"a159c706f31762a2c36dc0f646579da837818fdc448e4144d9f02417232173ca"} Oct 29 22:45:16.860982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:16.860148 4952 scope.go:122] "RemoveContainer" containerID="97029fec14c62832191a51e7432e5c8eff0e11fd5da7d3158c6f439d3c894dd9" Oct 29 22:45:16.860982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:16.860680 4952 scope.go:122] "RemoveContainer" containerID="a159c706f31762a2c36dc0f646579da837818fdc448e4144d9f02417232173ca" Oct 29 22:45:16.861279 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:16.861078 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888565-jhhc5_odh-ai-gateway-infra(a9ae5e2e-4abe-4ac4-8a68-0773b4098020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" Oct 29 22:45:16.865703 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:16.865638 4952 generic.go:356] "Generic (PLEG): container finished" podID="95346df9-7c90-4c49-817d-aac3d90ef04f" containerID="0fed92b19edd1c90c5abac72b36ac62e004c42640858d2ebe4f2fdbfd231f4ea" exitCode=0 Oct 29 22:45:16.865878 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:16.865856 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" event={"ID":"95346df9-7c90-4c49-817d-aac3d90ef04f","Type":"ContainerDied","Data":"0fed92b19edd1c90c5abac72b36ac62e004c42640858d2ebe4f2fdbfd231f4ea"} Oct 29 22:45:17.556624 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:17.555773 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:17.637007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:17.636972 4952 scope.go:122] "RemoveContainer" containerID="a796f6f1375c4a664f45d66ecb1174f4c1c8902d04ca74ec6d9eacd0ddbe01d7" Oct 29 22:45:17.637228 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:17.637068 4952 scope.go:122] "RemoveContainer" containerID="8e584c9abdd19fc8c2cd12387fa757aa3c09573751aefb6cec944c5bc51b7a2c" Oct 29 22:45:17.875941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:17.875872 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" event={"ID":"93a24bdb-4f05-4544-bd37-1896c18dffa9","Type":"ContainerStarted","Data":"e1c1763cd4d112c15e845f83e7fe4a969a48950202fd4c80209741a01ddc4ef7"} Oct 29 22:45:17.879436 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:17.879401 4952 generic.go:356] "Generic (PLEG): container finished" podID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerID="5bb553358997547e3c2e7ebfd42459adafca2f74a1b03566f2dd071fffb5df5b" exitCode=60 Oct 29 22:45:17.879605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:17.879491 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" event={"ID":"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c","Type":"ContainerDied","Data":"5bb553358997547e3c2e7ebfd42459adafca2f74a1b03566f2dd071fffb5df5b"} Oct 29 22:45:17.879605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:17.879554 4952 scope.go:122] "RemoveContainer" containerID="3460905d0550f14b451bc0989bd0fbc2dde9bade25e83fd2dc22433aaa707e77" Oct 29 22:45:17.880019 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:17.879997 4952 scope.go:122] "RemoveContainer" containerID="5bb553358997547e3c2e7ebfd42459adafca2f74a1b03566f2dd071fffb5df5b" Oct 29 22:45:17.880333 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:17.880301 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw_odh-ai-gateway-infra(7fc63e06-4fc4-4f20-ad12-1e3419d39a1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" Oct 29 22:45:17.881711 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:17.881683 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" event={"ID":"cce3d7ba-803e-45c6-a601-33268ea9223a","Type":"ContainerStarted","Data":"dd3ed6341753ec5379b83e730d6a21d632bb3e33d416785bcda133ec125246f0"} Oct 29 22:45:17.924275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:17.923830 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5"] Oct 29 22:45:18.112302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.112268 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:18.135465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.135439 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" Oct 29 22:45:18.240171 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.240134 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-kube-api-access-7lsbn\" (UniqueName: \"kubernetes.io/projected/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-kube-api-access-7lsbn\") pod \"a9ae5e2e-4abe-4ac4-8a68-0773b4098020\" (UID: \"a9ae5e2e-4abe-4ac4-8a68-0773b4098020\") " Oct 29 22:45:18.240171 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.240175 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-openshift-ca\") pod \"95346df9-7c90-4c49-817d-aac3d90ef04f\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " Oct 29 22:45:18.240466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.240293 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-maas-api-service-ca\") pod \"a9ae5e2e-4abe-4ac4-8a68-0773b4098020\" (UID: \"a9ae5e2e-4abe-4ac4-8a68-0773b4098020\") " Oct 29 22:45:18.240466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.240342 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-xks-ca\") pod \"95346df9-7c90-4c49-817d-aac3d90ef04f\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " Oct 29 22:45:18.240599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.240525 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95346df9-7c90-4c49-817d-aac3d90ef04f-kube-api-access-6c44p\" (UniqueName: \"kubernetes.io/projected/95346df9-7c90-4c49-817d-aac3d90ef04f-kube-api-access-6c44p\") pod \"95346df9-7c90-4c49-817d-aac3d90ef04f\" (UID: \"95346df9-7c90-4c49-817d-aac3d90ef04f\") " Oct 29 22:45:18.240839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.240638 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-openshift-ca" pod "95346df9-7c90-4c49-817d-aac3d90ef04f" (UID: "95346df9-7c90-4c49-817d-aac3d90ef04f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:18.241010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.240943 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-maas-api-service-ca" pod "a9ae5e2e-4abe-4ac4-8a68-0773b4098020" (UID: "a9ae5e2e-4abe-4ac4-8a68-0773b4098020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:18.241098 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.241081 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-openshift-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.241167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.241101 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.243276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.243250 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-xks-ca" pod "95346df9-7c90-4c49-817d-aac3d90ef04f" (UID: "95346df9-7c90-4c49-817d-aac3d90ef04f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:18.243539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.243498 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95346df9-7c90-4c49-817d-aac3d90ef04f-kube-api-access-6c44p" pod "95346df9-7c90-4c49-817d-aac3d90ef04f" (UID: "95346df9-7c90-4c49-817d-aac3d90ef04f"). InnerVolumeSpecName "kube-api-access-6c44p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:18.243641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.243558 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-kube-api-access-7lsbn" pod "a9ae5e2e-4abe-4ac4-8a68-0773b4098020" (UID: "a9ae5e2e-4abe-4ac4-8a68-0773b4098020"). InnerVolumeSpecName "kube-api-access-7lsbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:18.342673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.342579 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lsbn\" (UniqueName: \"kubernetes.io/projected/a9ae5e2e-4abe-4ac4-8a68-0773b4098020-kube-api-access-7lsbn\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.342673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.342616 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95346df9-7c90-4c49-817d-aac3d90ef04f-maas-api-xks-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.342673 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.342632 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c44p\" (UniqueName: \"kubernetes.io/projected/95346df9-7c90-4c49-817d-aac3d90ef04f-kube-api-access-6c44p\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.407896 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.407852 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" podUID="72a61920-3f20-4e73-84f8-dbafb64381f1" containerName="main" containerID="cri-o://b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf" gracePeriod=45 Oct 29 22:45:18.637847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.637260 4952 scope.go:122] "RemoveContainer" containerID="d7f0dc8da67b5b72b3e9e8d593e6ea4042a8fa3cea3932545252aa028ab96750" Oct 29 22:45:18.637847 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.637461 4952 scope.go:122] "RemoveContainer" containerID="16961e4cfb4d1793f52e1b00c45c62991c672e15f32108553c652e7b12da6e14" Oct 29 22:45:18.735718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.735691 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:45:18.849154 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.849118 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a61920-3f20-4e73-84f8-dbafb64381f1-kube-api-access-k24kc\" (UniqueName: \"kubernetes.io/projected/72a61920-3f20-4e73-84f8-dbafb64381f1-kube-api-access-k24kc\") pod \"72a61920-3f20-4e73-84f8-dbafb64381f1\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " Oct 29 22:45:18.849362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.849172 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-dshm\") pod \"72a61920-3f20-4e73-84f8-dbafb64381f1\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " Oct 29 22:45:18.849362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.849271 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a61920-3f20-4e73-84f8-dbafb64381f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/72a61920-3f20-4e73-84f8-dbafb64381f1-tls-certs\") pod \"72a61920-3f20-4e73-84f8-dbafb64381f1\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " Oct 29 22:45:18.849362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.849327 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-model-cache\") pod \"72a61920-3f20-4e73-84f8-dbafb64381f1\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " Oct 29 22:45:18.849535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.849376 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-tmp-dir\") pod \"72a61920-3f20-4e73-84f8-dbafb64381f1\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " Oct 29 22:45:18.849535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.849444 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-home\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-home\") pod \"72a61920-3f20-4e73-84f8-dbafb64381f1\" (UID: \"72a61920-3f20-4e73-84f8-dbafb64381f1\") " Oct 29 22:45:18.849797 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.849768 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-model-cache" pod "72a61920-3f20-4e73-84f8-dbafb64381f1" (UID: "72a61920-3f20-4e73-84f8-dbafb64381f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:18.849935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.849891 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-tmp-dir" pod "72a61920-3f20-4e73-84f8-dbafb64381f1" (UID: "72a61920-3f20-4e73-84f8-dbafb64381f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:18.850030 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.849999 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-home" pod "72a61920-3f20-4e73-84f8-dbafb64381f1" (UID: "72a61920-3f20-4e73-84f8-dbafb64381f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:18.850164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.850116 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.850164 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.850140 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.892376 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.892276 4952 generic.go:356] "Generic (PLEG): container finished" podID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerID="e81102d735deb7dc1b4015c4782e81186681b516450a17c31f21036207f7855f" exitCode=60 Oct 29 22:45:18.892604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.892431 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" event={"ID":"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788","Type":"ContainerDied","Data":"e81102d735deb7dc1b4015c4782e81186681b516450a17c31f21036207f7855f"} Oct 29 22:45:18.892604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.892494 4952 scope.go:122] "RemoveContainer" containerID="8e584c9abdd19fc8c2cd12387fa757aa3c09573751aefb6cec944c5bc51b7a2c" Oct 29 22:45:18.893037 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.893013 4952 scope.go:122] "RemoveContainer" containerID="e81102d735deb7dc1b4015c4782e81186681b516450a17c31f21036207f7855f" Oct 29 22:45:18.893528 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:18.893497 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr_odh-ai-gateway-infra(b01ec1be-c7c4-4c50-bcb5-67e4b33ab788)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" Oct 29 22:45:18.896642 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.896614 4952 generic.go:356] "Generic (PLEG): container finished" podID="72a61920-3f20-4e73-84f8-dbafb64381f1" containerID="b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf" exitCode=0 Oct 29 22:45:18.896750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.896646 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" event={"ID":"72a61920-3f20-4e73-84f8-dbafb64381f1","Type":"ContainerDied","Data":"b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf"} Oct 29 22:45:18.896750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.896681 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" Oct 29 22:45:18.896750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.896692 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" event={"ID":"72a61920-3f20-4e73-84f8-dbafb64381f1","Type":"ContainerDied","Data":"59fa0b7f53af483dcdb350c9d21e6b326443abaf832c7522653e9dd3dd03bb0f"} Oct 29 22:45:18.899500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.899472 4952 generic.go:356] "Generic (PLEG): container finished" podID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerID="e1c1763cd4d112c15e845f83e7fe4a969a48950202fd4c80209741a01ddc4ef7" exitCode=60 Oct 29 22:45:18.899604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.899542 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" event={"ID":"93a24bdb-4f05-4544-bd37-1896c18dffa9","Type":"ContainerDied","Data":"e1c1763cd4d112c15e845f83e7fe4a969a48950202fd4c80209741a01ddc4ef7"} Oct 29 22:45:18.899987 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.899965 4952 scope.go:122] "RemoveContainer" containerID="e1c1763cd4d112c15e845f83e7fe4a969a48950202fd4c80209741a01ddc4ef7" Oct 29 22:45:18.900329 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:18.900271 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x_odh-ai-gateway-infra(93a24bdb-4f05-4544-bd37-1896c18dffa9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" Oct 29 22:45:18.902654 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.901936 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" event={"ID":"037a603f-52cf-4222-a5be-57f8cb73dc98","Type":"ContainerStarted","Data":"4e4eda24051ccfffdd0ddfe1ae4cc31a02204a430ec5f9910e0fe15a59d05f2d"} Oct 29 22:45:18.903708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.903670 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" event={"ID":"a9ae5e2e-4abe-4ac4-8a68-0773b4098020","Type":"ContainerDied","Data":"12ce90537599c7788754db9186eab46fa6e1cdae6b637279a889e450d5a0b570"} Oct 29 22:45:18.904068 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.903699 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5" Oct 29 22:45:18.911095 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.911040 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" Oct 29 22:45:18.911309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.911045 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg" event={"ID":"95346df9-7c90-4c49-817d-aac3d90ef04f","Type":"ContainerDied","Data":"4f374f144c62e4b0d47511bfa2450d1083f90848dd040646b9b7b88dbb8cae61"} Oct 29 22:45:18.911442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.911338 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f374f144c62e4b0d47511bfa2450d1083f90848dd040646b9b7b88dbb8cae61" Oct 29 22:45:18.913519 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.913494 4952 generic.go:356] "Generic (PLEG): container finished" podID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerID="dd3ed6341753ec5379b83e730d6a21d632bb3e33d416785bcda133ec125246f0" exitCode=60 Oct 29 22:45:18.913708 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.913562 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" event={"ID":"cce3d7ba-803e-45c6-a601-33268ea9223a","Type":"ContainerDied","Data":"dd3ed6341753ec5379b83e730d6a21d632bb3e33d416785bcda133ec125246f0"} Oct 29 22:45:18.913955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.913932 4952 scope.go:122] "RemoveContainer" containerID="dd3ed6341753ec5379b83e730d6a21d632bb3e33d416785bcda133ec125246f0" Oct 29 22:45:18.914283 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:18.914257 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx_odh-ai-gateway-infra(cce3d7ba-803e-45c6-a601-33268ea9223a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" Oct 29 22:45:18.943720 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.943661 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-dshm" pod "72a61920-3f20-4e73-84f8-dbafb64381f1" (UID: "72a61920-3f20-4e73-84f8-dbafb64381f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:18.944821 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.944763 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a61920-3f20-4e73-84f8-dbafb64381f1-kube-api-access-k24kc" pod "72a61920-3f20-4e73-84f8-dbafb64381f1" (UID: "72a61920-3f20-4e73-84f8-dbafb64381f1"). InnerVolumeSpecName "kube-api-access-k24kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:18.945552 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.945109 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a61920-3f20-4e73-84f8-dbafb64381f1-tls-certs" pod "72a61920-3f20-4e73-84f8-dbafb64381f1" (UID: "72a61920-3f20-4e73-84f8-dbafb64381f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:18.952301 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.951444 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.952301 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.951476 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k24kc\" (UniqueName: \"kubernetes.io/projected/72a61920-3f20-4e73-84f8-dbafb64381f1-kube-api-access-k24kc\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.952301 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.951487 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a61920-3f20-4e73-84f8-dbafb64381f1-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.952301 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.951497 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a61920-3f20-4e73-84f8-dbafb64381f1-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.956082 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.956052 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x"] Oct 29 22:45:18.992229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:18.992190 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw"] Oct 29 22:45:19.040793 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.040755 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx"] Oct 29 22:45:19.067373 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.067345 4952 scope.go:122] "RemoveContainer" containerID="b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf" Oct 29 22:45:19.095813 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.095742 4952 scope.go:122] "RemoveContainer" containerID="b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf" Oct 29 22:45:19.097212 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:19.096300 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf\": container with ID starting with b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf not found: ID does not exist" containerID="b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf" Oct 29 22:45:19.097727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.097370 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf"} err="failed to get container status \"b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf\": rpc error: code = NotFound desc = could not find container \"b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf\": container with ID starting with b08edb1e3c60f0886fa298ea60e2dd4d2e36cf4345d53e93b9fdca8da7111baf not found: ID does not exist" Oct 29 22:45:19.097727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.097409 4952 scope.go:122] "RemoveContainer" containerID="70e548bc3538ec09387a4536916b22c471d3111a01b74e54a22be9b37f698664" Oct 29 22:45:19.115816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.115792 4952 scope.go:122] "RemoveContainer" containerID="a159c706f31762a2c36dc0f646579da837818fdc448e4144d9f02417232173ca" Oct 29 22:45:19.129526 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.129498 4952 scope.go:122] "RemoveContainer" containerID="8e30c643597f8b87a57aca375d98b34c4e03f143c0d815b528e4b98d055e1955" Oct 29 22:45:19.236356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.236311 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" Oct 29 22:45:19.243293 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.243264 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5"] Oct 29 22:45:19.247758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.247732 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-jhhc5"] Oct 29 22:45:19.333282 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.333237 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm"] Oct 29 22:45:19.339146 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.339111 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm"] Oct 29 22:45:19.351182 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.351128 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:19.357238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.357208 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-kube-api-access-nx2fc\" (UniqueName: \"kubernetes.io/projected/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-kube-api-access-nx2fc\") pod \"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c\" (UID: \"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c\") " Oct 29 22:45:19.357377 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.357324 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-maas-api-service-ca\") pod \"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c\" (UID: \"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c\") " Oct 29 22:45:19.357744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.357715 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-maas-api-service-ca" pod "7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" (UID: "7fc63e06-4fc4-4f20-ad12-1e3419d39a1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:19.357882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.357854 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.359517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.359493 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-kube-api-access-nx2fc" pod "7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" (UID: "7fc63e06-4fc4-4f20-ad12-1e3419d39a1c"). InnerVolumeSpecName "kube-api-access-nx2fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:19.459237 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.459149 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx2fc\" (UniqueName: \"kubernetes.io/projected/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c-kube-api-access-nx2fc\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:19.632841 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.632787 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d09411/test-model-d09411-kserve-d5c87d9df-vjltm" podUID="72a61920-3f20-4e73-84f8-dbafb64381f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:19.681281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.681250 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:19.926016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.925562 4952 generic.go:356] "Generic (PLEG): container finished" podID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerID="4e4eda24051ccfffdd0ddfe1ae4cc31a02204a430ec5f9910e0fe15a59d05f2d" exitCode=60 Oct 29 22:45:19.926016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.925671 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" event={"ID":"037a603f-52cf-4222-a5be-57f8cb73dc98","Type":"ContainerDied","Data":"4e4eda24051ccfffdd0ddfe1ae4cc31a02204a430ec5f9910e0fe15a59d05f2d"} Oct 29 22:45:19.926016 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.925769 4952 scope.go:122] "RemoveContainer" containerID="a796f6f1375c4a664f45d66ecb1174f4c1c8902d04ca74ec6d9eacd0ddbe01d7" Oct 29 22:45:19.927761 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.926763 4952 scope.go:122] "RemoveContainer" containerID="4e4eda24051ccfffdd0ddfe1ae4cc31a02204a430ec5f9910e0fe15a59d05f2d" Oct 29 22:45:19.927761 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:19.927131 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk_odh-ai-gateway-infra(037a603f-52cf-4222-a5be-57f8cb73dc98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" Oct 29 22:45:19.944075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.937645 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" event={"ID":"7fc63e06-4fc4-4f20-ad12-1e3419d39a1c","Type":"ContainerDied","Data":"c002aa5d2de7ecf9634c6988eb46a4f406a1b1ecd1479122a6733f07b223fcac"} Oct 29 22:45:19.944075 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.937775 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw" Oct 29 22:45:19.958071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.957929 4952 scope.go:122] "RemoveContainer" containerID="5bb553358997547e3c2e7ebfd42459adafca2f74a1b03566f2dd071fffb5df5b" Oct 29 22:45:19.968286 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.968254 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr"] Oct 29 22:45:19.974325 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.974286 4952 generic.go:356] "Generic (PLEG): container finished" podID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerID="ade98275a62ca8ed21d5ed6ed41226294432f5b029ff48be297d1feed64a41dd" exitCode=60 Oct 29 22:45:19.974426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.974401 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" event={"ID":"713cded9-f796-464a-8fed-59dfbcedb3d7","Type":"ContainerDied","Data":"ade98275a62ca8ed21d5ed6ed41226294432f5b029ff48be297d1feed64a41dd"} Oct 29 22:45:19.974829 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.974809 4952 scope.go:122] "RemoveContainer" containerID="ade98275a62ca8ed21d5ed6ed41226294432f5b029ff48be297d1feed64a41dd" Oct 29 22:45:19.975179 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:19.975157 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p_odh-ai-gateway-infra(713cded9-f796-464a-8fed-59dfbcedb3d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" Oct 29 22:45:19.978213 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.978182 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" event={"ID":"6fec9dea-1cd0-41f4-93c1-f19617b84ca1","Type":"ContainerStarted","Data":"c0957e1a10b6e53b2a5fadf5d357f6f99ea838c07367b25f631dd8c7829af0e0"} Oct 29 22:45:19.979393 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:19.979367 4952 scope.go:122] "RemoveContainer" containerID="d7f0dc8da67b5b72b3e9e8d593e6ea4042a8fa3cea3932545252aa028ab96750" Oct 29 22:45:20.071079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.067810 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw"] Oct 29 22:45:20.071079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.070671 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d09411-29888565-rcrbw"] Oct 29 22:45:20.150480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.150436 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk"] Oct 29 22:45:20.287103 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.287070 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" Oct 29 22:45:20.290806 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.290773 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" Oct 29 22:45:20.294959 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.294935 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" Oct 29 22:45:20.373100 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.372998 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-kube-api-access-q87dc\" (UniqueName: \"kubernetes.io/projected/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-kube-api-access-q87dc\") pod \"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788\" (UID: \"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788\") " Oct 29 22:45:20.373262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.373103 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a24bdb-4f05-4544-bd37-1896c18dffa9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a24bdb-4f05-4544-bd37-1896c18dffa9-maas-api-service-ca\") pod \"93a24bdb-4f05-4544-bd37-1896c18dffa9\" (UID: \"93a24bdb-4f05-4544-bd37-1896c18dffa9\") " Oct 29 22:45:20.373305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.373257 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce3d7ba-803e-45c6-a601-33268ea9223a-kube-api-access-xqhzk\" (UniqueName: \"kubernetes.io/projected/cce3d7ba-803e-45c6-a601-33268ea9223a-kube-api-access-xqhzk\") pod \"cce3d7ba-803e-45c6-a601-33268ea9223a\" (UID: \"cce3d7ba-803e-45c6-a601-33268ea9223a\") " Oct 29 22:45:20.373305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.373295 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce3d7ba-803e-45c6-a601-33268ea9223a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce3d7ba-803e-45c6-a601-33268ea9223a-maas-api-service-ca\") pod \"cce3d7ba-803e-45c6-a601-33268ea9223a\" (UID: \"cce3d7ba-803e-45c6-a601-33268ea9223a\") " Oct 29 22:45:20.373381 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.373335 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-maas-api-service-ca\") pod \"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788\" (UID: \"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788\") " Oct 29 22:45:20.373381 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.373369 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a24bdb-4f05-4544-bd37-1896c18dffa9-kube-api-access-8k9kf\" (UniqueName: \"kubernetes.io/projected/93a24bdb-4f05-4544-bd37-1896c18dffa9-kube-api-access-8k9kf\") pod \"93a24bdb-4f05-4544-bd37-1896c18dffa9\" (UID: \"93a24bdb-4f05-4544-bd37-1896c18dffa9\") " Oct 29 22:45:20.373647 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.373561 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a24bdb-4f05-4544-bd37-1896c18dffa9-maas-api-service-ca" pod "93a24bdb-4f05-4544-bd37-1896c18dffa9" (UID: "93a24bdb-4f05-4544-bd37-1896c18dffa9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:20.373756 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.373732 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce3d7ba-803e-45c6-a601-33268ea9223a-maas-api-service-ca" pod "cce3d7ba-803e-45c6-a601-33268ea9223a" (UID: "cce3d7ba-803e-45c6-a601-33268ea9223a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:20.374042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.374021 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a24bdb-4f05-4544-bd37-1896c18dffa9-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.374250 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.374120 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-maas-api-service-ca" pod "b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" (UID: "b01ec1be-c7c4-4c50-bcb5-67e4b33ab788"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:20.374481 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.374232 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce3d7ba-803e-45c6-a601-33268ea9223a-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.375805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.375782 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-kube-api-access-q87dc" pod "b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" (UID: "b01ec1be-c7c4-4c50-bcb5-67e4b33ab788"). InnerVolumeSpecName "kube-api-access-q87dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:20.375988 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.375838 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a24bdb-4f05-4544-bd37-1896c18dffa9-kube-api-access-8k9kf" pod "93a24bdb-4f05-4544-bd37-1896c18dffa9" (UID: "93a24bdb-4f05-4544-bd37-1896c18dffa9"). InnerVolumeSpecName "kube-api-access-8k9kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:20.376804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.376744 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce3d7ba-803e-45c6-a601-33268ea9223a-kube-api-access-xqhzk" pod "cce3d7ba-803e-45c6-a601-33268ea9223a" (UID: "cce3d7ba-803e-45c6-a601-33268ea9223a"). InnerVolumeSpecName "kube-api-access-xqhzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:20.476334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.476289 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k9kf\" (UniqueName: \"kubernetes.io/projected/93a24bdb-4f05-4544-bd37-1896c18dffa9-kube-api-access-8k9kf\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.476334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.476337 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q87dc\" (UniqueName: \"kubernetes.io/projected/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-kube-api-access-q87dc\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.476534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.476356 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqhzk\" (UniqueName: \"kubernetes.io/projected/cce3d7ba-803e-45c6-a601-33268ea9223a-kube-api-access-xqhzk\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.476534 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.476372 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:20.645501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.645410 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a61920-3f20-4e73-84f8-dbafb64381f1" path="/var/lib/kubelet/pods/72a61920-3f20-4e73-84f8-dbafb64381f1/volumes" Oct 29 22:45:20.646003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.645979 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" path="/var/lib/kubelet/pods/7fc63e06-4fc4-4f20-ad12-1e3419d39a1c/volumes" Oct 29 22:45:20.646478 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.646457 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" path="/var/lib/kubelet/pods/a9ae5e2e-4abe-4ac4-8a68-0773b4098020/volumes" Oct 29 22:45:20.988757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.988677 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" Oct 29 22:45:20.988757 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.988711 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x" event={"ID":"93a24bdb-4f05-4544-bd37-1896c18dffa9","Type":"ContainerDied","Data":"4cc21d1e0f0a9dd29341adcd6f9630b126d898f5da834e468dc014f3a75e5ef0"} Oct 29 22:45:20.989273 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.988776 4952 scope.go:122] "RemoveContainer" containerID="e1c1763cd4d112c15e845f83e7fe4a969a48950202fd4c80209741a01ddc4ef7" Oct 29 22:45:20.991010 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.990982 4952 generic.go:356] "Generic (PLEG): container finished" podID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerID="c0957e1a10b6e53b2a5fadf5d357f6f99ea838c07367b25f631dd8c7829af0e0" exitCode=60 Oct 29 22:45:20.991163 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.991059 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" event={"ID":"6fec9dea-1cd0-41f4-93c1-f19617b84ca1","Type":"ContainerDied","Data":"c0957e1a10b6e53b2a5fadf5d357f6f99ea838c07367b25f631dd8c7829af0e0"} Oct 29 22:45:20.991703 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.991499 4952 scope.go:122] "RemoveContainer" containerID="c0957e1a10b6e53b2a5fadf5d357f6f99ea838c07367b25f631dd8c7829af0e0" Oct 29 22:45:20.991853 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:20.991825 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj_odh-ai-gateway-infra(6fec9dea-1cd0-41f4-93c1-f19617b84ca1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" Oct 29 22:45:20.995470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.995452 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" Oct 29 22:45:20.995592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.995476 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx" event={"ID":"cce3d7ba-803e-45c6-a601-33268ea9223a","Type":"ContainerDied","Data":"5b293712fb7c6ee3d32bf43e39daed7196208b02cdc1bd1050be1a9e848a1337"} Oct 29 22:45:20.997239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.997215 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" event={"ID":"b01ec1be-c7c4-4c50-bcb5-67e4b33ab788","Type":"ContainerDied","Data":"214ebb72e11f651b656f11198cd6d97bd0280540e6d670d3fc789e4dcb9e9992"} Oct 29 22:45:20.997337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:20.997222 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr" Oct 29 22:45:21.001712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.001691 4952 scope.go:122] "RemoveContainer" containerID="16961e4cfb4d1793f52e1b00c45c62991c672e15f32108553c652e7b12da6e14" Oct 29 22:45:21.018327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.018299 4952 scope.go:122] "RemoveContainer" containerID="dd3ed6341753ec5379b83e730d6a21d632bb3e33d416785bcda133ec125246f0" Oct 29 22:45:21.034353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.034329 4952 scope.go:122] "RemoveContainer" containerID="e81102d735deb7dc1b4015c4782e81186681b516450a17c31f21036207f7855f" Oct 29 22:45:21.048306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.047916 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx"] Oct 29 22:45:21.053774 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.053742 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-07e166-29888565-lgrqx"] Oct 29 22:45:21.150939 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.150717 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x"] Oct 29 22:45:21.155730 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.155694 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-201a3d-29888565-5x47x"] Oct 29 22:45:21.188408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.188341 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj"] Oct 29 22:45:21.205097 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.200036 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:21.206106 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.206078 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr"] Oct 29 22:45:21.214958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.214394 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" Oct 29 22:45:21.214958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.214471 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f52917-29888565-scvtr"] Oct 29 22:45:21.296432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.296351 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037a603f-52cf-4222-a5be-57f8cb73dc98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037a603f-52cf-4222-a5be-57f8cb73dc98-maas-api-service-ca\") pod \"037a603f-52cf-4222-a5be-57f8cb73dc98\" (UID: \"037a603f-52cf-4222-a5be-57f8cb73dc98\") " Oct 29 22:45:21.296432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.296386 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037a603f-52cf-4222-a5be-57f8cb73dc98-kube-api-access-hvqsq\" (UniqueName: \"kubernetes.io/projected/037a603f-52cf-4222-a5be-57f8cb73dc98-kube-api-access-hvqsq\") pod \"037a603f-52cf-4222-a5be-57f8cb73dc98\" (UID: \"037a603f-52cf-4222-a5be-57f8cb73dc98\") " Oct 29 22:45:21.296816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.296792 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037a603f-52cf-4222-a5be-57f8cb73dc98-maas-api-service-ca" pod "037a603f-52cf-4222-a5be-57f8cb73dc98" (UID: "037a603f-52cf-4222-a5be-57f8cb73dc98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.298714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.298686 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037a603f-52cf-4222-a5be-57f8cb73dc98-kube-api-access-hvqsq" pod "037a603f-52cf-4222-a5be-57f8cb73dc98" (UID: "037a603f-52cf-4222-a5be-57f8cb73dc98"). InnerVolumeSpecName "kube-api-access-hvqsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.318248 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.318217 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p"] Oct 29 22:45:21.397569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.397531 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037a603f-52cf-4222-a5be-57f8cb73dc98-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.397569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.397565 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvqsq\" (UniqueName: \"kubernetes.io/projected/037a603f-52cf-4222-a5be-57f8cb73dc98-kube-api-access-hvqsq\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.506830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.506802 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" Oct 29 22:45:21.599634 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.599595 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713cded9-f796-464a-8fed-59dfbcedb3d7-kube-api-access-c4td9\" (UniqueName: \"kubernetes.io/projected/713cded9-f796-464a-8fed-59dfbcedb3d7-kube-api-access-c4td9\") pod \"713cded9-f796-464a-8fed-59dfbcedb3d7\" (UID: \"713cded9-f796-464a-8fed-59dfbcedb3d7\") " Oct 29 22:45:21.599830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.599644 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713cded9-f796-464a-8fed-59dfbcedb3d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713cded9-f796-464a-8fed-59dfbcedb3d7-maas-api-service-ca\") pod \"713cded9-f796-464a-8fed-59dfbcedb3d7\" (UID: \"713cded9-f796-464a-8fed-59dfbcedb3d7\") " Oct 29 22:45:21.600087 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.600056 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713cded9-f796-464a-8fed-59dfbcedb3d7-maas-api-service-ca" pod "713cded9-f796-464a-8fed-59dfbcedb3d7" (UID: "713cded9-f796-464a-8fed-59dfbcedb3d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.600212 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.600197 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713cded9-f796-464a-8fed-59dfbcedb3d7-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.601881 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.601852 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713cded9-f796-464a-8fed-59dfbcedb3d7-kube-api-access-c4td9" pod "713cded9-f796-464a-8fed-59dfbcedb3d7" (UID: "713cded9-f796-464a-8fed-59dfbcedb3d7"). InnerVolumeSpecName "kube-api-access-c4td9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.701402 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:21.701367 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4td9\" (UniqueName: \"kubernetes.io/projected/713cded9-f796-464a-8fed-59dfbcedb3d7-kube-api-access-c4td9\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:22.011843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.011754 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" Oct 29 22:45:22.012423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.011774 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p" event={"ID":"713cded9-f796-464a-8fed-59dfbcedb3d7","Type":"ContainerDied","Data":"fde77886f7ac125027820999c4a987cb3424ac5a333d4eca79b27cd38ff976e5"} Oct 29 22:45:22.012423 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.011898 4952 scope.go:122] "RemoveContainer" containerID="ade98275a62ca8ed21d5ed6ed41226294432f5b029ff48be297d1feed64a41dd" Oct 29 22:45:22.016432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.016401 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" event={"ID":"037a603f-52cf-4222-a5be-57f8cb73dc98","Type":"ContainerDied","Data":"fb9872f398c1e373233e941e16e19f478eec9093919e56c9e2dd5b85a5685020"} Oct 29 22:45:22.016568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.016418 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk" Oct 29 22:45:22.026420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.026379 4952 scope.go:122] "RemoveContainer" containerID="4e4eda24051ccfffdd0ddfe1ae4cc31a02204a430ec5f9910e0fe15a59d05f2d" Oct 29 22:45:22.154824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.153066 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p"] Oct 29 22:45:22.154824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.153480 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-91520c-29888565-b7s5p"] Oct 29 22:45:22.187480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.187450 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk"] Oct 29 22:45:22.192066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.192038 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-749f72-29888565-cmlnk"] Oct 29 22:45:22.205188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.205163 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" Oct 29 22:45:22.316038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.315951 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-kube-api-access-7vpk5\" (UniqueName: \"kubernetes.io/projected/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-kube-api-access-7vpk5\") pod \"6fec9dea-1cd0-41f4-93c1-f19617b84ca1\" (UID: \"6fec9dea-1cd0-41f4-93c1-f19617b84ca1\") " Oct 29 22:45:22.316038 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.316031 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-maas-api-service-ca\") pod \"6fec9dea-1cd0-41f4-93c1-f19617b84ca1\" (UID: \"6fec9dea-1cd0-41f4-93c1-f19617b84ca1\") " Oct 29 22:45:22.316442 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.316416 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-maas-api-service-ca" pod "6fec9dea-1cd0-41f4-93c1-f19617b84ca1" (UID: "6fec9dea-1cd0-41f4-93c1-f19617b84ca1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:22.318228 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.318207 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-kube-api-access-7vpk5" pod "6fec9dea-1cd0-41f4-93c1-f19617b84ca1" (UID: "6fec9dea-1cd0-41f4-93c1-f19617b84ca1"). InnerVolumeSpecName "kube-api-access-7vpk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:22.417508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.417465 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vpk5\" (UniqueName: \"kubernetes.io/projected/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-kube-api-access-7vpk5\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:22.417508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.417501 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fec9dea-1cd0-41f4-93c1-f19617b84ca1-maas-api-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:22.644634 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.644596 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" path="/var/lib/kubelet/pods/037a603f-52cf-4222-a5be-57f8cb73dc98/volumes" Oct 29 22:45:22.645201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.645181 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" path="/var/lib/kubelet/pods/713cded9-f796-464a-8fed-59dfbcedb3d7/volumes" Oct 29 22:45:22.645633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.645617 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" path="/var/lib/kubelet/pods/93a24bdb-4f05-4544-bd37-1896c18dffa9/volumes" Oct 29 22:45:22.646432 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.646412 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" path="/var/lib/kubelet/pods/b01ec1be-c7c4-4c50-bcb5-67e4b33ab788/volumes" Oct 29 22:45:22.646814 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.646799 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" path="/var/lib/kubelet/pods/cce3d7ba-803e-45c6-a601-33268ea9223a/volumes" Oct 29 22:45:22.911975 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:22.911879 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:23.028174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:23.028129 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" event={"ID":"6fec9dea-1cd0-41f4-93c1-f19617b84ca1","Type":"ContainerDied","Data":"86642b8e87e2ca67683a5c4c272690f44e92f31cb2a17ad059cbc2d78916d204"} Oct 29 22:45:23.028174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:23.028157 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj" Oct 29 22:45:23.028761 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:23.028208 4952 scope.go:122] "RemoveContainer" containerID="c0957e1a10b6e53b2a5fadf5d357f6f99ea838c07367b25f631dd8c7829af0e0" Oct 29 22:45:23.087824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:23.087789 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj"] Oct 29 22:45:23.090623 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:23.090596 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0c3d9a-29888565-kghjj"] Oct 29 22:45:24.421633 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:24.421596 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:24.644015 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:24.643977 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" path="/var/lib/kubelet/pods/6fec9dea-1cd0-41f4-93c1-f19617b84ca1/volumes" Oct 29 22:45:24.874990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:24.874951 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:25.045973 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:25.045934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:26.031607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:26.031572 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:27.626893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:27.626845 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:27.947827 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:27.947736 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt"] Oct 29 22:45:27.948330 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:27.948292 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" containerID="cri-o://9f3cab521de89ded609030fe41192da60e992b3286b71cf33d89c1b1813f79f0" gracePeriod=30 Oct 29 22:45:28.624020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:28.623982 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:29.225559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:29.225514 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:30.555255 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:30.555215 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:30.724602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:30.724565 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:30.991051 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:30.991013 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:31.296077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:31.295992 4952 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:45:31.296077 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:31.296064 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:45:32.394320 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:32.394281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:33.113876 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.113835 4952 generic.go:356] "Generic (PLEG): container finished" podID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerID="9f3cab521de89ded609030fe41192da60e992b3286b71cf33d89c1b1813f79f0" exitCode=0 Oct 29 22:45:33.114094 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.113936 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" event={"ID":"44576234-7466-4ad4-9001-7c9b153fdcd3","Type":"ContainerDied","Data":"9f3cab521de89ded609030fe41192da60e992b3286b71cf33d89c1b1813f79f0"} Oct 29 22:45:33.656425 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.656395 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:45:33.658831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.658796 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-envoy\") pod \"44576234-7466-4ad4-9001-7c9b153fdcd3\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " Oct 29 22:45:33.659008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.658884 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-data\") pod \"44576234-7466-4ad4-9001-7c9b153fdcd3\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " Oct 29 22:45:33.659008 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.658957 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44576234-7466-4ad4-9001-7c9b153fdcd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44576234-7466-4ad4-9001-7c9b153fdcd3-istiod-ca-cert\") pod \"44576234-7466-4ad4-9001-7c9b153fdcd3\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " Oct 29 22:45:33.659150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659022 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-certs\") pod \"44576234-7466-4ad4-9001-7c9b153fdcd3\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " Oct 29 22:45:33.659150 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659110 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-credential-socket\") pod \"44576234-7466-4ad4-9001-7c9b153fdcd3\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " Oct 29 22:45:33.659262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659159 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-kube-api-access-k5tjw\" (UniqueName: \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-kube-api-access-k5tjw\") pod \"44576234-7466-4ad4-9001-7c9b153fdcd3\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " Oct 29 22:45:33.659644 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659249 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-data" pod "44576234-7466-4ad4-9001-7c9b153fdcd3" (UID: "44576234-7466-4ad4-9001-7c9b153fdcd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:33.659784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659382 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-certs" pod "44576234-7466-4ad4-9001-7c9b153fdcd3" (UID: "44576234-7466-4ad4-9001-7c9b153fdcd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:33.659784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659492 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-credential-socket" pod "44576234-7466-4ad4-9001-7c9b153fdcd3" (UID: "44576234-7466-4ad4-9001-7c9b153fdcd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:33.659784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659653 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-socket\") pod \"44576234-7466-4ad4-9001-7c9b153fdcd3\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " Oct 29 22:45:33.659784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659722 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-token\" (UniqueName: \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-token\") pod \"44576234-7466-4ad4-9001-7c9b153fdcd3\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " Oct 29 22:45:33.659784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659756 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-podinfo\") pod \"44576234-7466-4ad4-9001-7c9b153fdcd3\" (UID: \"44576234-7466-4ad4-9001-7c9b153fdcd3\") " Oct 29 22:45:33.660115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659577 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44576234-7466-4ad4-9001-7c9b153fdcd3-istiod-ca-cert" pod "44576234-7466-4ad4-9001-7c9b153fdcd3" (UID: "44576234-7466-4ad4-9001-7c9b153fdcd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:33.660115 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.659855 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-socket" pod "44576234-7466-4ad4-9001-7c9b153fdcd3" (UID: "44576234-7466-4ad4-9001-7c9b153fdcd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:33.660230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.660174 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:33.660230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.660194 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:33.660230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.660209 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:33.660230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.660222 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:33.660447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.660236 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44576234-7466-4ad4-9001-7c9b153fdcd3-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:33.662501 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.662472 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-envoy" pod "44576234-7466-4ad4-9001-7c9b153fdcd3" (UID: "44576234-7466-4ad4-9001-7c9b153fdcd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:33.662616 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.662515 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-podinfo" pod "44576234-7466-4ad4-9001-7c9b153fdcd3" (UID: "44576234-7466-4ad4-9001-7c9b153fdcd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:33.662616 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.662551 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-token" pod "44576234-7466-4ad4-9001-7c9b153fdcd3" (UID: "44576234-7466-4ad4-9001-7c9b153fdcd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:33.662616 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.662575 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-kube-api-access-k5tjw" pod "44576234-7466-4ad4-9001-7c9b153fdcd3" (UID: "44576234-7466-4ad4-9001-7c9b153fdcd3"). InnerVolumeSpecName "kube-api-access-k5tjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:33.761351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.761265 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tjw\" (UniqueName: \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-kube-api-access-k5tjw\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:33.761351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.761298 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:33.761351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.761310 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:33.761351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:33.761318 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44576234-7466-4ad4-9001-7c9b153fdcd3-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:34.042102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:34.042009 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:34.122812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:34.122780 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" Oct 29 22:45:34.123042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:34.122817 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt" event={"ID":"44576234-7466-4ad4-9001-7c9b153fdcd3","Type":"ContainerDied","Data":"b56e152b9c712db1a7d828719829ebd1a042622fc82eeefb77356cdf00ec427e"} Oct 29 22:45:34.123042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:34.122938 4952 scope.go:122] "RemoveContainer" containerID="9f3cab521de89ded609030fe41192da60e992b3286b71cf33d89c1b1813f79f0" Oct 29 22:45:34.181479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:34.181442 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt"] Oct 29 22:45:34.194786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:34.194755 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f52917-openshift-default-7ff96d4d97-4hjmt"] Oct 29 22:45:34.646511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:34.646474 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" path="/var/lib/kubelet/pods/44576234-7466-4ad4-9001-7c9b153fdcd3/volumes" Oct 29 22:45:35.624977 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:35.624129 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:36.163686 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:36.163629 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:45:37.241551 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:37.241508 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:38.501891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:38.499112 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk"] Oct 29 22:45:38.501891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:38.499368 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" podUID="2cf522b0-f6c8-40c3-8775-50ff08781fc1" containerName="maas-api" containerID="cri-o://dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79" gracePeriod=30 Oct 29 22:45:38.619588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:38.619545 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:38.984846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:38.984732 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:39.086507 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.086477 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:45:39.169020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.153763 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:39.169020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.165364 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf522b0-f6c8-40c3-8775-50ff08781fc1-kube-api-access-2d6dt\" (UniqueName: \"kubernetes.io/projected/2cf522b0-f6c8-40c3-8775-50ff08781fc1-kube-api-access-2d6dt\") pod \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " Oct 29 22:45:39.169020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.165507 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-maas-api-tls\") pod \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " Oct 29 22:45:39.169020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.165562 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-metrics-certs\") pod \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " Oct 29 22:45:39.169020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.165631 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf522b0-f6c8-40c3-8775-50ff08781fc1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf522b0-f6c8-40c3-8775-50ff08781fc1-openshift-service-ca\") pod \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\" (UID: \"2cf522b0-f6c8-40c3-8775-50ff08781fc1\") " Oct 29 22:45:39.169020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.166278 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf522b0-f6c8-40c3-8775-50ff08781fc1-openshift-service-ca" pod "2cf522b0-f6c8-40c3-8775-50ff08781fc1" (UID: "2cf522b0-f6c8-40c3-8775-50ff08781fc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:39.171061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.170970 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-metrics-certs" pod "2cf522b0-f6c8-40c3-8775-50ff08781fc1" (UID: "2cf522b0-f6c8-40c3-8775-50ff08781fc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:39.171061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.171004 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf522b0-f6c8-40c3-8775-50ff08781fc1-kube-api-access-2d6dt" pod "2cf522b0-f6c8-40c3-8775-50ff08781fc1" (UID: "2cf522b0-f6c8-40c3-8775-50ff08781fc1"). InnerVolumeSpecName "kube-api-access-2d6dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:39.171394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.171373 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-maas-api-tls" pod "2cf522b0-f6c8-40c3-8775-50ff08781fc1" (UID: "2cf522b0-f6c8-40c3-8775-50ff08781fc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:39.178968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.178935 4952 generic.go:356] "Generic (PLEG): container finished" podID="2cf522b0-f6c8-40c3-8775-50ff08781fc1" containerID="dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79" exitCode=0 Oct 29 22:45:39.179113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.179016 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" event={"ID":"2cf522b0-f6c8-40c3-8775-50ff08781fc1","Type":"ContainerDied","Data":"dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79"} Oct 29 22:45:39.179113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.179068 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" event={"ID":"2cf522b0-f6c8-40c3-8775-50ff08781fc1","Type":"ContainerDied","Data":"a9f1aec62f5d8fb6d9092a18044100ee79b8b513c6b8fc637ee513f6fc98984c"} Oct 29 22:45:39.179113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.179100 4952 scope.go:122] "RemoveContainer" containerID="dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79" Oct 29 22:45:39.179292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.179258 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk" Oct 29 22:45:39.206074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.206046 4952 scope.go:122] "RemoveContainer" containerID="dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79" Oct 29 22:45:39.206376 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:39.206353 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79\": container with ID starting with dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79 not found: ID does not exist" containerID="dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79" Oct 29 22:45:39.206453 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.206384 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79"} err="failed to get container status \"dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79\": rpc error: code = NotFound desc = could not find container \"dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79\": container with ID starting with dd92e3896270a37b91dff6db55e23c0d1fca06b509db6854f7ba3fc6234b7b79 not found: ID does not exist" Oct 29 22:45:39.246675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.246640 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk"] Oct 29 22:45:39.261311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.261253 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09411-6bd98df45-l4rvk"] Oct 29 22:45:39.282648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.273744 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:39.282648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.273783 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf522b0-f6c8-40c3-8775-50ff08781fc1-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:39.282648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.273800 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf522b0-f6c8-40c3-8775-50ff08781fc1-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:39.282648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.273815 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d6dt\" (UniqueName: \"kubernetes.io/projected/2cf522b0-f6c8-40c3-8775-50ff08781fc1-kube-api-access-2d6dt\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:39.332131 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.331696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:39.595811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.595727 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:39.668724 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.668682 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg"] Oct 29 22:45:39.678057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.678014 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4afaba-jdhsg"] Oct 29 22:45:39.804985 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:39.804414 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:40.309660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:40.309615 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:40.645776 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:40.645740 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf522b0-f6c8-40c3-8775-50ff08781fc1" path="/var/lib/kubelet/pods/2cf522b0-f6c8-40c3-8775-50ff08781fc1/volumes" Oct 29 22:45:40.646313 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:40.646294 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95346df9-7c90-4c49-817d-aac3d90ef04f" path="/var/lib/kubelet/pods/95346df9-7c90-4c49-817d-aac3d90ef04f/volumes" Oct 29 22:45:41.457504 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:41.457460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:43.113042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:43.113001 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:44.754306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:44.754265 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:45.528437 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:45.528395 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:45:46.331355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:46.331309 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:46.523372 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:46.523330 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:46.695744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:46.695654 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:46.797215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:46.797172 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq"] Oct 29 22:45:46.797464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:46.797432 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerName="istio-proxy" containerID="cri-o://836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40" gracePeriod=30 Oct 29 22:45:46.923809 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:46.923769 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:48.106758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:48.106716 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:48.317469 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:48.317430 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:48.488602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:48.488519 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:48.690875 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:48.690836 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:49.162205 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162165 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp"] Oct 29 22:45:49.162702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162677 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerName="cleanup" Oct 29 22:45:49.162702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162696 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerName="cleanup" Oct 29 22:45:49.162702 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162709 4952 cpu_manager.go:414] "containerMap: removing container" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162714 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162721 4952 cpu_manager.go:414] "containerMap: removing container" podUID="95346df9-7c90-4c49-817d-aac3d90ef04f" containerName="revoke-keys" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162727 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95346df9-7c90-4c49-817d-aac3d90ef04f" containerName="revoke-keys" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162735 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162740 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162747 4952 cpu_manager.go:414] "containerMap: removing container" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162753 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162764 4952 cpu_manager.go:414] "containerMap: removing container" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162771 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162782 4952 cpu_manager.go:414] "containerMap: removing container" podUID="2cf522b0-f6c8-40c3-8775-50ff08781fc1" containerName="maas-api" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162789 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf522b0-f6c8-40c3-8775-50ff08781fc1" containerName="maas-api" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162800 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162807 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162818 4952 cpu_manager.go:414] "containerMap: removing container" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162825 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162837 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162844 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162861 4952 cpu_manager.go:414] "containerMap: removing container" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162868 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162884 4952 cpu_manager.go:414] "containerMap: removing container" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162916 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162931 4952 cpu_manager.go:414] "containerMap: removing container" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162939 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162953 4952 cpu_manager.go:414] "containerMap: removing container" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerName="cleanup" Oct 29 22:45:49.162957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162962 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162978 4952 cpu_manager.go:414] "containerMap: removing container" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.162986 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163009 4952 cpu_manager.go:414] "containerMap: removing container" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163018 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163035 4952 cpu_manager.go:414] "containerMap: removing container" podUID="72a61920-3f20-4e73-84f8-dbafb64381f1" containerName="main" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163044 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a61920-3f20-4e73-84f8-dbafb64381f1" containerName="main" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163056 4952 cpu_manager.go:414] "containerMap: removing container" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163063 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163073 4952 cpu_manager.go:414] "containerMap: removing container" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163080 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163091 4952 cpu_manager.go:414] "containerMap: removing container" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163098 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163123 4952 cpu_manager.go:414] "containerMap: removing container" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163127 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163136 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163144 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163156 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163164 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163309 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163320 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163327 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163333 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163339 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163345 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163352 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163358 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163364 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a61920-3f20-4e73-84f8-dbafb64381f1" containerName="main" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163370 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163376 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163382 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163389 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163397 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163404 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163410 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163416 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="95346df9-7c90-4c49-817d-aac3d90ef04f" containerName="revoke-keys" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163423 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerName="cleanup" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163430 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="44576234-7466-4ad4-9001-7c9b153fdcd3" containerName="istio-proxy" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163436 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf522b0-f6c8-40c3-8775-50ff08781fc1" containerName="maas-api" Oct 29 22:45:49.163739 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.163442 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerName="cleanup" Oct 29 22:45:49.168410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.168389 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.186513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.186481 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp"] Oct 29 22:45:49.203957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.197169 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:45:49.203957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.197251 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:45:49.299105 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.299071 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4lm\" (UniqueName: \"kubernetes.io/projected/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-kube-api-access-vn4lm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.299295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.299217 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.299374 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.299359 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.400338 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.400296 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.400569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.400404 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.400569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.400443 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4lm\" (UniqueName: \"kubernetes.io/projected/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-kube-api-access-vn4lm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.401174 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.401149 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.403109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.403081 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.426741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.426668 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4lm\" (UniqueName: \"kubernetes.io/projected/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-kube-api-access-vn4lm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.484306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.484264 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:49.805339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.805296 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp"] Oct 29 22:45:49.808029 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:45:49.807978 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb33613f_1bfc_487d_b68c_a7f1592d2b6f.slice/crio-df6b80c218a8e86f5b4e927cc8f4975175774fa4c29ee900c9dae8ee008f2a93 WatchSource:0}: Error finding container df6b80c218a8e86f5b4e927cc8f4975175774fa4c29ee900c9dae8ee008f2a93: Status 404 returned error can't find the container with id df6b80c218a8e86f5b4e927cc8f4975175774fa4c29ee900c9dae8ee008f2a93 Oct 29 22:45:49.852646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:49.852064 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:50.260584 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:50.260530 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" event={"ID":"bb33613f-1bfc-487d-b68c-a7f1592d2b6f","Type":"ContainerStarted","Data":"df9230330c762f0721bfb8614196602305a56161779c2fc9aeeff30d66f5f1c4"} Oct 29 22:45:50.261033 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:50.260644 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" event={"ID":"bb33613f-1bfc-487d-b68c-a7f1592d2b6f","Type":"ContainerStarted","Data":"df6b80c218a8e86f5b4e927cc8f4975175774fa4c29ee900c9dae8ee008f2a93"} Oct 29 22:45:50.304850 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:50.304777 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" podStartSLOduration=1.3047578 podStartE2EDuration="1.3047578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:50.303472346 +0000 UTC m=+3815.778060764" watchObservedRunningTime="2026-10-29 22:45:50.3047578 +0000 UTC m=+3815.779346218" Oct 29 22:45:51.270820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:51.270785 4952 generic.go:356] "Generic (PLEG): container finished" podID="bb33613f-1bfc-487d-b68c-a7f1592d2b6f" containerID="df9230330c762f0721bfb8614196602305a56161779c2fc9aeeff30d66f5f1c4" exitCode=0 Oct 29 22:45:51.271240 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:51.270882 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" event={"ID":"bb33613f-1bfc-487d-b68c-a7f1592d2b6f","Type":"ContainerDied","Data":"df9230330c762f0721bfb8614196602305a56161779c2fc9aeeff30d66f5f1c4"} Oct 29 22:45:51.518923 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:51.518871 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:52.101835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.101804 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:45:52.236284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236246 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-token\" (UniqueName: \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-token\") pod \"1ccaa807-dd67-431a-8b48-07cf649eefed\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " Oct 29 22:45:52.236515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236305 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-podinfo\") pod \"1ccaa807-dd67-431a-8b48-07cf649eefed\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " Oct 29 22:45:52.236515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236351 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-socket\") pod \"1ccaa807-dd67-431a-8b48-07cf649eefed\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " Oct 29 22:45:52.236515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236431 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-certs\") pod \"1ccaa807-dd67-431a-8b48-07cf649eefed\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " Oct 29 22:45:52.236515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236481 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-data\") pod \"1ccaa807-dd67-431a-8b48-07cf649eefed\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " Oct 29 22:45:52.236515 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236512 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-envoy\") pod \"1ccaa807-dd67-431a-8b48-07cf649eefed\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " Oct 29 22:45:52.236812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236609 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ccaa807-dd67-431a-8b48-07cf649eefed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ccaa807-dd67-431a-8b48-07cf649eefed-istiod-ca-cert\") pod \"1ccaa807-dd67-431a-8b48-07cf649eefed\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " Oct 29 22:45:52.236812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236695 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-credential-socket\") pod \"1ccaa807-dd67-431a-8b48-07cf649eefed\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " Oct 29 22:45:52.236812 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236760 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-kube-api-access-zdrz4\" (UniqueName: \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-kube-api-access-zdrz4\") pod \"1ccaa807-dd67-431a-8b48-07cf649eefed\" (UID: \"1ccaa807-dd67-431a-8b48-07cf649eefed\") " Oct 29 22:45:52.236996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236692 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-socket" pod "1ccaa807-dd67-431a-8b48-07cf649eefed" (UID: "1ccaa807-dd67-431a-8b48-07cf649eefed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:52.236996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236884 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-certs" pod "1ccaa807-dd67-431a-8b48-07cf649eefed" (UID: "1ccaa807-dd67-431a-8b48-07cf649eefed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:52.236996 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.236831 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-data" pod "1ccaa807-dd67-431a-8b48-07cf649eefed" (UID: "1ccaa807-dd67-431a-8b48-07cf649eefed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:52.237485 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.237235 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.237485 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.237264 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.237485 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.237280 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.237485 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.237443 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-credential-socket" pod "1ccaa807-dd67-431a-8b48-07cf649eefed" (UID: "1ccaa807-dd67-431a-8b48-07cf649eefed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:52.238222 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.238199 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ccaa807-dd67-431a-8b48-07cf649eefed-istiod-ca-cert" pod "1ccaa807-dd67-431a-8b48-07cf649eefed" (UID: "1ccaa807-dd67-431a-8b48-07cf649eefed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:52.239142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.239117 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-token" pod "1ccaa807-dd67-431a-8b48-07cf649eefed" (UID: "1ccaa807-dd67-431a-8b48-07cf649eefed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:52.239142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.239133 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-podinfo" pod "1ccaa807-dd67-431a-8b48-07cf649eefed" (UID: "1ccaa807-dd67-431a-8b48-07cf649eefed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:52.239548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.239528 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-envoy" pod "1ccaa807-dd67-431a-8b48-07cf649eefed" (UID: "1ccaa807-dd67-431a-8b48-07cf649eefed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:52.239602 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.239575 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-kube-api-access-zdrz4" pod "1ccaa807-dd67-431a-8b48-07cf649eefed" (UID: "1ccaa807-dd67-431a-8b48-07cf649eefed"). InnerVolumeSpecName "kube-api-access-zdrz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:52.279968 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.279928 4952 generic.go:356] "Generic (PLEG): container finished" podID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerID="836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40" exitCode=0 Oct 29 22:45:52.280403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.279974 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" event={"ID":"1ccaa807-dd67-431a-8b48-07cf649eefed","Type":"ContainerDied","Data":"836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40"} Oct 29 22:45:52.280403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.279999 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" Oct 29 22:45:52.280403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.280037 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq" event={"ID":"1ccaa807-dd67-431a-8b48-07cf649eefed","Type":"ContainerDied","Data":"b595be7358b2c57679709f55eb1b68e293fce178342cb93d72b149dc7a062824"} Oct 29 22:45:52.280403 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.280065 4952 scope.go:122] "RemoveContainer" containerID="836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40" Oct 29 22:45:52.293826 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.293803 4952 scope.go:122] "RemoveContainer" containerID="836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40" Oct 29 22:45:52.294235 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:45:52.294211 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40\": container with ID starting with 836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40 not found: ID does not exist" containerID="836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40" Oct 29 22:45:52.294360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.294241 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40"} err="failed to get container status \"836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40\": rpc error: code = NotFound desc = could not find container \"836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40\": container with ID starting with 836d9287347151f77bcb685a32a15b3120529ef2bd14c0a3ea92f77c736b7d40 not found: ID does not exist" Oct 29 22:45:52.338275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.338236 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ccaa807-dd67-431a-8b48-07cf649eefed-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.338275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.338268 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.338275 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.338281 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdrz4\" (UniqueName: \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-kube-api-access-zdrz4\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.338580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.338292 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.338580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.338301 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.338580 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.338309 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ccaa807-dd67-431a-8b48-07cf649eefed-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.343236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.341008 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq"] Oct 29 22:45:52.351300 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.351271 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d09411-openshift-default-8d6c66fd5-lb6nq"] Oct 29 22:45:52.439648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.439613 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:52.540158 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.540109 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-openshift-ca\") pod \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " Oct 29 22:45:52.540331 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.540214 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-xks-ca\") pod \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " Oct 29 22:45:52.540331 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.540312 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-kube-api-access-vn4lm\" (UniqueName: \"kubernetes.io/projected/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-kube-api-access-vn4lm\") pod \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\" (UID: \"bb33613f-1bfc-487d-b68c-a7f1592d2b6f\") " Oct 29 22:45:52.540562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.540533 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-openshift-ca" pod "bb33613f-1bfc-487d-b68c-a7f1592d2b6f" (UID: "bb33613f-1bfc-487d-b68c-a7f1592d2b6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:52.540729 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.540706 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-openshift-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.542637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.542606 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-xks-ca" pod "bb33613f-1bfc-487d-b68c-a7f1592d2b6f" (UID: "bb33613f-1bfc-487d-b68c-a7f1592d2b6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:52.542764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.542658 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-kube-api-access-vn4lm" pod "bb33613f-1bfc-487d-b68c-a7f1592d2b6f" (UID: "bb33613f-1bfc-487d-b68c-a7f1592d2b6f"). InnerVolumeSpecName "kube-api-access-vn4lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:52.642362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.642276 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-maas-api-xks-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.642362 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.642313 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4lm\" (UniqueName: \"kubernetes.io/projected/bb33613f-1bfc-487d-b68c-a7f1592d2b6f-kube-api-access-vn4lm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:45:52.645803 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:52.645761 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" path="/var/lib/kubelet/pods/1ccaa807-dd67-431a-8b48-07cf649eefed/volumes" Oct 29 22:45:53.119039 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:53.119003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:53.289818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:53.289784 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" event={"ID":"bb33613f-1bfc-487d-b68c-a7f1592d2b6f","Type":"ContainerDied","Data":"df6b80c218a8e86f5b4e927cc8f4975175774fa4c29ee900c9dae8ee008f2a93"} Oct 29 22:45:53.289818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:53.289822 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp" Oct 29 22:45:53.290335 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:53.289837 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6b80c218a8e86f5b4e927cc8f4975175774fa4c29ee900c9dae8ee008f2a93" Oct 29 22:45:54.395177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:54.395137 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:54.833349 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:54.833307 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:56.515770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:56.515659 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:58.118024 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:58.117978 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:58.287262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:58.287215 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:58.458242 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:58.458140 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:59.847252 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:45:59.847218 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:00.021431 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:00.021389 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:01.497204 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:01.497156 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:03.023920 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:03.023876 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:04.602330 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:04.602292 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:05.300548 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:05.300503 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:05.447993 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:05.447947 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:46:05.471873 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:05.471832 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:06.202727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:06.202692 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:07.580117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:07.580079 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:08.059446 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:08.059363 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:08.131843 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:08.131808 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp"] Oct 29 22:46:08.144687 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:08.144654 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f52917-v5jtp"] Oct 29 22:46:08.234864 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:08.234819 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:08.643972 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:08.643930 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb33613f-1bfc-487d-b68c-a7f1592d2b6f" path="/var/lib/kubelet/pods/bb33613f-1bfc-487d-b68c-a7f1592d2b6f/volumes" Oct 29 22:46:08.744672 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:08.744626 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:09.920990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:09.920953 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:11.289202 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:11.289159 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:11.798604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:11.798567 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:11.976576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:11.976530 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:12.478175 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:12.478139 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:12.644635 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:12.644595 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:13.605526 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:13.605488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:14.180860 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:14.180822 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:14.347467 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:14.347424 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:15.204346 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:15.204303 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:15.527335 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:46:15.527253 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:46:16.102951 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:16.102878 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:16.273784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:16.273748 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:16.520595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:16.520490 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:16.702748 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:16.702701 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:16.882663 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:16.882627 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:17.064859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:17.064818 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:17.256072 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:17.256035 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:17.440605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:17.440518 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:17.907629 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:17.907592 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:19.075698 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:19.075663 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:19.858668 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:19.858633 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:20.049001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:20.048961 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:20.260846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:20.260759 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:20.587261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:20.587224 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:46:20.725239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:20.725200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:21.963935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:21.963874 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z"] Oct 29 22:46:21.977950 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:21.977914 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk"] Oct 29 22:46:22.133608 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:22.133571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:22.513597 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:22.513563 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:24.118701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:24.118658 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:25.512695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:25.512654 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs"] Oct 29 22:46:25.513230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:25.512989 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" containerID="cri-o://c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54" gracePeriod=30 Oct 29 22:46:25.525370 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:25.521512 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:25.734926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:25.729713 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:25.915310 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:25.915270 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:27.485820 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:27.485784 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:28.015116 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:28.015076 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7"] Oct 29 22:46:28.198521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:28.198481 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:28.366724 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:28.366683 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:29.226073 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:29.226033 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:30.800653 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.800627 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:46:30.837306 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837253 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-socket\") pod \"85d2c654-5589-4747-986c-94ec4fd16ffd\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " Oct 29 22:46:30.837465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837332 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d2c654-5589-4747-986c-94ec4fd16ffd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d2c654-5589-4747-986c-94ec4fd16ffd-istiod-ca-cert\") pod \"85d2c654-5589-4747-986c-94ec4fd16ffd\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " Oct 29 22:46:30.837465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837392 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-envoy\") pod \"85d2c654-5589-4747-986c-94ec4fd16ffd\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " Oct 29 22:46:30.837465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837419 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-certs\") pod \"85d2c654-5589-4747-986c-94ec4fd16ffd\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " Oct 29 22:46:30.837465 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837452 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-kube-api-access-rfw5r\" (UniqueName: \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-kube-api-access-rfw5r\") pod \"85d2c654-5589-4747-986c-94ec4fd16ffd\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " Oct 29 22:46:30.837697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837510 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-podinfo\") pod \"85d2c654-5589-4747-986c-94ec4fd16ffd\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " Oct 29 22:46:30.837697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837565 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-credential-socket\") pod \"85d2c654-5589-4747-986c-94ec4fd16ffd\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " Oct 29 22:46:30.837697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837581 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-socket" pod "85d2c654-5589-4747-986c-94ec4fd16ffd" (UID: "85d2c654-5589-4747-986c-94ec4fd16ffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:30.837697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837626 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-token\" (UniqueName: \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-token\") pod \"85d2c654-5589-4747-986c-94ec4fd16ffd\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " Oct 29 22:46:30.837957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837725 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-data\") pod \"85d2c654-5589-4747-986c-94ec4fd16ffd\" (UID: \"85d2c654-5589-4747-986c-94ec4fd16ffd\") " Oct 29 22:46:30.837957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837775 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d2c654-5589-4747-986c-94ec4fd16ffd-istiod-ca-cert" pod "85d2c654-5589-4747-986c-94ec4fd16ffd" (UID: "85d2c654-5589-4747-986c-94ec4fd16ffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:30.837957 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.837783 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-certs" pod "85d2c654-5589-4747-986c-94ec4fd16ffd" (UID: "85d2c654-5589-4747-986c-94ec4fd16ffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:30.838162 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.838049 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-credential-socket" pod "85d2c654-5589-4747-986c-94ec4fd16ffd" (UID: "85d2c654-5589-4747-986c-94ec4fd16ffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:30.838227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.838182 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.838227 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.838201 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d2c654-5589-4747-986c-94ec4fd16ffd-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.838340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.838215 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.838340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.838251 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.838340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.838306 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-data" pod "85d2c654-5589-4747-986c-94ec4fd16ffd" (UID: "85d2c654-5589-4747-986c-94ec4fd16ffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:30.840187 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.840158 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-token" pod "85d2c654-5589-4747-986c-94ec4fd16ffd" (UID: "85d2c654-5589-4747-986c-94ec4fd16ffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:30.840638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.840607 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-envoy" pod "85d2c654-5589-4747-986c-94ec4fd16ffd" (UID: "85d2c654-5589-4747-986c-94ec4fd16ffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:30.840744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.840660 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-podinfo" pod "85d2c654-5589-4747-986c-94ec4fd16ffd" (UID: "85d2c654-5589-4747-986c-94ec4fd16ffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:30.840744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.840705 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-kube-api-access-rfw5r" pod "85d2c654-5589-4747-986c-94ec4fd16ffd" (UID: "85d2c654-5589-4747-986c-94ec4fd16ffd"). InnerVolumeSpecName "kube-api-access-rfw5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:30.939652 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.939612 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.939652 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.939654 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfw5r\" (UniqueName: \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-kube-api-access-rfw5r\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.939868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.939670 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.939868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.939683 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:30.939868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:30.939696 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d2c654-5589-4747-986c-94ec4fd16ffd-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:31.040387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.040292 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:31.476312 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.476273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:31.608610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.608578 4952 generic.go:356] "Generic (PLEG): container finished" podID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerID="c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54" exitCode=0 Oct 29 22:46:31.608818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.608626 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" Oct 29 22:46:31.608818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.608668 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" event={"ID":"85d2c654-5589-4747-986c-94ec4fd16ffd","Type":"ContainerDied","Data":"c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54"} Oct 29 22:46:31.608818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.608731 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs" event={"ID":"85d2c654-5589-4747-986c-94ec4fd16ffd","Type":"ContainerDied","Data":"56a5485600df780ac84815be68ec64d360b4f0cca29261c014d69dcdf258ab90"} Oct 29 22:46:31.608818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.608762 4952 scope.go:122] "RemoveContainer" containerID="c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54" Oct 29 22:46:31.621576 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.621555 4952 scope.go:122] "RemoveContainer" containerID="dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8" Oct 29 22:46:31.634849 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.634655 4952 scope.go:122] "RemoveContainer" containerID="c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54" Oct 29 22:46:31.635037 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:46:31.635015 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54\": container with ID starting with c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54 not found: ID does not exist" containerID="c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54" Oct 29 22:46:31.635122 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.635046 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54"} err="failed to get container status \"c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54\": rpc error: code = NotFound desc = could not find container \"c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54\": container with ID starting with c64d2a6a9ebac630b485ce9f4101006f9786815280538d00a445cef9904e5a54 not found: ID does not exist" Oct 29 22:46:31.635122 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.635075 4952 scope.go:122] "RemoveContainer" containerID="dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8" Oct 29 22:46:31.635353 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:46:31.635327 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8\": container with ID starting with dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8 not found: ID does not exist" containerID="dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8" Oct 29 22:46:31.635466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.635358 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8"} err="failed to get container status \"dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8\": rpc error: code = NotFound desc = could not find container \"dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8\": container with ID starting with dfb3432d4ac6f784defe1cb08d4244b04baf5ff7b3715e9f689f6da403f1eea8 not found: ID does not exist" Oct 29 22:46:31.650268 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.650246 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:31.680765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.680735 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs"] Oct 29 22:46:31.694198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:31.694174 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-27b6db-openshift-default-5bb795dcb9-g2wqs"] Oct 29 22:46:32.046264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:32.046224 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:32.644197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:32.644164 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" path="/var/lib/kubelet/pods/85d2c654-5589-4747-986c-94ec4fd16ffd/volumes" Oct 29 22:46:32.764373 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:32.764321 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:33.015431 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:33.015326 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" podUID="1ab0a623-d106-4608-817a-6445d1d0cf39" containerName="payload-pre-processing" containerID="cri-o://95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19" gracePeriod=25 Oct 29 22:46:33.282867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:33.282781 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" podUID="1ab0a623-d106-4608-817a-6445d1d0cf39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:34.349718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:34.349680 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:34.875160 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:34.875116 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:35.053260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:35.053218 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:35.551577 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:35.551543 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:36.559547 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:36.559500 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:36.611575 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:36.611538 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm"] Oct 29 22:46:36.637433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:36.637398 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789"] Oct 29 22:46:36.748760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:36.748710 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:36.918553 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:36.918514 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:36.986229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:36.986190 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" containerID="cri-o://20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce" gracePeriod=45 Oct 29 22:46:37.001535 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.001498 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" containerID="cri-o://6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a" gracePeriod=45 Oct 29 22:46:37.292061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.292032 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 22:46:37.315750 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.315713 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:37.316200 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.316179 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.414516 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tls-certs\" (UniqueName: \"kubernetes.io/secret/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tls-certs\") pod \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.414655 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tmp-dir\") pod \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.414717 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-model-cache\") pod \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.414795 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tmp-dir\") pod \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.414832 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-dshm\") pod \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.414878 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tls-certs\") pod \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.414947 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-home\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-home\") pod \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.415017 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-kube-api-access-d67wv\" (UniqueName: \"kubernetes.io/projected/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-kube-api-access-d67wv\") pod \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.415076 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-home\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-home\") pod \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.415118 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-dshm\") pod \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.415167 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-model-cache\") pod \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\" (UID: \"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.415213 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-kube-api-access-9mhvx\" (UniqueName: \"kubernetes.io/projected/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-kube-api-access-9mhvx\") pod \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\" (UID: \"5da6a393-1fd5-4ab4-b273-c6d32454aa0c\") " Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.415456 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tmp-dir" pod "5da6a393-1fd5-4ab4-b273-c6d32454aa0c" (UID: "5da6a393-1fd5-4ab4-b273-c6d32454aa0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.415699 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.415527 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-model-cache" pod "5da6a393-1fd5-4ab4-b273-c6d32454aa0c" (UID: "5da6a393-1fd5-4ab4-b273-c6d32454aa0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.416075 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-home" pod "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" (UID: "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.416245 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tmp-dir" pod "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" (UID: "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.416368 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-home" pod "5da6a393-1fd5-4ab4-b273-c6d32454aa0c" (UID: "5da6a393-1fd5-4ab4-b273-c6d32454aa0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.417284 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.416498 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-model-cache" pod "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" (UID: "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.420346 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.419813 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-kube-api-access-d67wv" pod "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" (UID: "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134"). InnerVolumeSpecName "kube-api-access-d67wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:37.420697 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.420672 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-dshm" pod "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" (UID: "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.421371 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.421345 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-dshm" pod "5da6a393-1fd5-4ab4-b273-c6d32454aa0c" (UID: "5da6a393-1fd5-4ab4-b273-c6d32454aa0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:37.421536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.421513 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-kube-api-access-9mhvx" pod "5da6a393-1fd5-4ab4-b273-c6d32454aa0c" (UID: "5da6a393-1fd5-4ab4-b273-c6d32454aa0c"). InnerVolumeSpecName "kube-api-access-9mhvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:37.421758 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.421728 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tls-certs" pod "5da6a393-1fd5-4ab4-b273-c6d32454aa0c" (UID: "5da6a393-1fd5-4ab4-b273-c6d32454aa0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:37.422441 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.422385 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tls-certs" pod "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" (UID: "92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:46:37.516461 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516430 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d67wv\" (UniqueName: \"kubernetes.io/projected/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-kube-api-access-d67wv\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.516461 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516458 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.516461 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516467 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.516727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516477 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.516727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516492 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mhvx\" (UniqueName: \"kubernetes.io/projected/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-kube-api-access-9mhvx\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.516727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516501 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.516727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516509 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.516727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516516 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.516727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516524 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.516727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516531 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da6a393-1fd5-4ab4-b273-c6d32454aa0c-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.516727 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.516538 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:37.664417 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.664384 4952 generic.go:356] "Generic (PLEG): container finished" podID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerID="20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce" exitCode=0 Oct 29 22:46:37.664967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.664455 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" Oct 29 22:46:37.664967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.664473 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" event={"ID":"5da6a393-1fd5-4ab4-b273-c6d32454aa0c","Type":"ContainerDied","Data":"20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce"} Oct 29 22:46:37.664967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.664534 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z" event={"ID":"5da6a393-1fd5-4ab4-b273-c6d32454aa0c","Type":"ContainerDied","Data":"880097e6df7c5b0238ce1fad0971ead7d23d4c7ce376f781a7f6469b46c9b9f3"} Oct 29 22:46:37.664967 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.664558 4952 scope.go:122] "RemoveContainer" containerID="20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce" Oct 29 22:46:37.666705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.666674 4952 generic.go:356] "Generic (PLEG): container finished" podID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerID="6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a" exitCode=0 Oct 29 22:46:37.667003 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.666883 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" Oct 29 22:46:37.668394 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.668363 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" event={"ID":"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134","Type":"ContainerDied","Data":"6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a"} Oct 29 22:46:37.668527 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.668412 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk" event={"ID":"92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134","Type":"ContainerDied","Data":"ff286d422c2df76a98ca65fcbe8c4a3b7b5ae0196500b56c0c9f341c82119996"} Oct 29 22:46:37.677836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.677814 4952 scope.go:122] "RemoveContainer" containerID="20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce" Oct 29 22:46:37.678164 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:46:37.678145 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce\": container with ID starting with 20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce not found: ID does not exist" containerID="20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce" Oct 29 22:46:37.678238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.678177 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce"} err="failed to get container status \"20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce\": rpc error: code = NotFound desc = could not find container \"20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce\": container with ID starting with 20d3b88b11ea574cc4e298e6fe1e4099ff0a4a1a3f65443031faf2e4fdad5fce not found: ID does not exist" Oct 29 22:46:37.678238 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.678196 4952 scope.go:122] "RemoveContainer" containerID="6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a" Oct 29 22:46:37.689031 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.689008 4952 scope.go:122] "RemoveContainer" containerID="6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a" Oct 29 22:46:37.689351 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:46:37.689318 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a\": container with ID starting with 6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a not found: ID does not exist" containerID="6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a" Oct 29 22:46:37.689444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.689360 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a"} err="failed to get container status \"6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a\": rpc error: code = NotFound desc = could not find container \"6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a\": container with ID starting with 6ad3d4069db8c22480ea646b6432ced2590f3d29092d18cde5544f12f7cab10a not found: ID does not exist" Oct 29 22:46:37.723622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.723595 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z"] Oct 29 22:46:37.730061 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.730035 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/premium-w0-f3b067-kserve-7987b66d96-k8p7z"] Oct 29 22:46:37.769638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.769591 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk"] Oct 29 22:46:37.772126 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:37.772102 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3b067/simulator-w0-f3b067-kserve-6d859fb8fc-t65pk"] Oct 29 22:46:38.117744 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:38.117712 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:38.306299 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:38.306206 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:38.644060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:38.644028 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" path="/var/lib/kubelet/pods/5da6a393-1fd5-4ab4-b273-c6d32454aa0c/volumes" Oct 29 22:46:38.644480 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:38.644467 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" path="/var/lib/kubelet/pods/92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134/volumes" Oct 29 22:46:38.739997 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:38.739958 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:39.025592 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:39.025484 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:40.767060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:40.767026 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:41.612305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:41.612240 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" podUID="6e76b81a-f826-46f6-95c7-27ed04ded082" containerName="payload-processing" containerID="cri-o://2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7" gracePeriod=25 Oct 29 22:46:41.641751 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:41.641717 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" podUID="ad3b2165-83e9-4020-8159-f045ac452bd0" containerName="payload-pre-processing" containerID="cri-o://fe0951340f00dea00f2dd9d59a01089c5f23c2af3936c5fb19e94f268c9369ed" gracePeriod=25 Oct 29 22:46:41.882559 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:41.882529 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:46:41.965822 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:41.965783 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e76b81a-f826-46f6-95c7-27ed04ded082-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76b81a-f826-46f6-95c7-27ed04ded082-plugins-config-volume\") pod \"6e76b81a-f826-46f6-95c7-27ed04ded082\" (UID: \"6e76b81a-f826-46f6-95c7-27ed04ded082\") " Oct 29 22:46:41.966057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:41.966003 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e76b81a-f826-46f6-95c7-27ed04ded082-kube-api-access-wdtb2\" (UniqueName: \"kubernetes.io/projected/6e76b81a-f826-46f6-95c7-27ed04ded082-kube-api-access-wdtb2\") pod \"6e76b81a-f826-46f6-95c7-27ed04ded082\" (UID: \"6e76b81a-f826-46f6-95c7-27ed04ded082\") " Oct 29 22:46:41.966271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:41.966245 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e76b81a-f826-46f6-95c7-27ed04ded082-plugins-config-volume" pod "6e76b81a-f826-46f6-95c7-27ed04ded082" (UID: "6e76b81a-f826-46f6-95c7-27ed04ded082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:41.966431 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:41.966413 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76b81a-f826-46f6-95c7-27ed04ded082-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:41.968625 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:41.968601 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e76b81a-f826-46f6-95c7-27ed04ded082-kube-api-access-wdtb2" pod "6e76b81a-f826-46f6-95c7-27ed04ded082" (UID: "6e76b81a-f826-46f6-95c7-27ed04ded082"). InnerVolumeSpecName "kube-api-access-wdtb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:42.067128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.067078 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdtb2\" (UniqueName: \"kubernetes.io/projected/6e76b81a-f826-46f6-95c7-27ed04ded082-kube-api-access-wdtb2\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:42.418836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.418800 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:42.719460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.719378 4952 generic.go:356] "Generic (PLEG): container finished" podID="6e76b81a-f826-46f6-95c7-27ed04ded082" containerID="2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7" exitCode=0 Oct 29 22:46:42.719460 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.719422 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" Oct 29 22:46:42.719650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.719469 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" event={"ID":"6e76b81a-f826-46f6-95c7-27ed04ded082","Type":"ContainerDied","Data":"2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7"} Oct 29 22:46:42.719650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.719525 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm" event={"ID":"6e76b81a-f826-46f6-95c7-27ed04ded082","Type":"ContainerDied","Data":"93ae1c1e712722ddc05802279bd326bbab76c6f9244be069a3e9acf28bf5349e"} Oct 29 22:46:42.719650 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.719548 4952 scope.go:122] "RemoveContainer" containerID="2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7" Oct 29 22:46:42.729435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.729414 4952 scope.go:122] "RemoveContainer" containerID="2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7" Oct 29 22:46:42.729717 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:46:42.729686 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7\": container with ID starting with 2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7 not found: ID does not exist" containerID="2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7" Oct 29 22:46:42.729784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.729731 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7"} err="failed to get container status \"2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7\": rpc error: code = NotFound desc = could not find container \"2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7\": container with ID starting with 2078a02821d9767a756cba9a83e5ebaba0cfdc731face4951bb8b5204965c3e7 not found: ID does not exist" Oct 29 22:46:42.777491 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.777460 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm"] Oct 29 22:46:42.791267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.791222 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-580af2-85bd6778c4-nwlfm"] Oct 29 22:46:42.981934 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:42.981818 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:43.156144 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:43.156105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:43.283359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:43.283271 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" podUID="1ab0a623-d106-4608-817a-6445d1d0cf39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:43.407389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:43.407354 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:43.656689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:43.656650 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" podUID="ad3b2165-83e9-4020-8159-f045ac452bd0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:44.134271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:44.134233 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:44.319982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:44.319936 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:46:44.662367 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:44.662325 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e76b81a-f826-46f6-95c7-27ed04ded082" path="/var/lib/kubelet/pods/6e76b81a-f826-46f6-95c7-27ed04ded082/volumes" Oct 29 22:46:45.531676 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:46:45.531642 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:46:45.960536 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:45.960498 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:47.673235 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:47.673196 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:47.849241 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:47.849197 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:47.964079 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:47.963991 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v"] Oct 29 22:46:47.964257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:47.964214 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" containerID="cri-o://7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b" gracePeriod=30 Oct 29 22:46:48.039508 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:48.039457 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:49.218070 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:49.218028 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:49.784389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:49.784351 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:49.955181 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:49.955135 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:50.395188 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:50.395149 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:51.690824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:51.690780 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:53.310294 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.310243 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:53.313007 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.312979 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:46:53.347356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.347332 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 22:46:53.395406 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.395354 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-socket\") pod \"6862987c-7402-4134-a7a7-4a1663aed0de\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " Oct 29 22:46:53.395618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.395416 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-certs\") pod \"6862987c-7402-4134-a7a7-4a1663aed0de\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " Oct 29 22:46:53.395618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.395519 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-data\") pod \"6862987c-7402-4134-a7a7-4a1663aed0de\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " Oct 29 22:46:53.395618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.395555 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6862987c-7402-4134-a7a7-4a1663aed0de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6862987c-7402-4134-a7a7-4a1663aed0de-istio-podinfo\") pod \"6862987c-7402-4134-a7a7-4a1663aed0de\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " Oct 29 22:46:53.395618 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.395586 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6862987c-7402-4134-a7a7-4a1663aed0de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6862987c-7402-4134-a7a7-4a1663aed0de-istiod-ca-cert\") pod \"6862987c-7402-4134-a7a7-4a1663aed0de\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " Oct 29 22:46:53.395885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.395641 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-kube-api-access-fmzbt\" (UniqueName: \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-kube-api-access-fmzbt\") pod \"6862987c-7402-4134-a7a7-4a1663aed0de\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " Oct 29 22:46:53.395885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.395744 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-istio-token\" (UniqueName: \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-istio-token\") pod \"6862987c-7402-4134-a7a7-4a1663aed0de\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " Oct 29 22:46:53.395885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.395800 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-envoy\") pod \"6862987c-7402-4134-a7a7-4a1663aed0de\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " Oct 29 22:46:53.395885 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.395842 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-credential-socket\") pod \"6862987c-7402-4134-a7a7-4a1663aed0de\" (UID: \"6862987c-7402-4134-a7a7-4a1663aed0de\") " Oct 29 22:46:53.396466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.396394 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-certs" pod "6862987c-7402-4134-a7a7-4a1663aed0de" (UID: "6862987c-7402-4134-a7a7-4a1663aed0de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.396466 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.396420 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-credential-socket" pod "6862987c-7402-4134-a7a7-4a1663aed0de" (UID: "6862987c-7402-4134-a7a7-4a1663aed0de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.396649 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.396617 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-socket" pod "6862987c-7402-4134-a7a7-4a1663aed0de" (UID: "6862987c-7402-4134-a7a7-4a1663aed0de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.396998 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.396973 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6862987c-7402-4134-a7a7-4a1663aed0de-istiod-ca-cert" pod "6862987c-7402-4134-a7a7-4a1663aed0de" (UID: "6862987c-7402-4134-a7a7-4a1663aed0de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:53.397099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.396979 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-data" pod "6862987c-7402-4134-a7a7-4a1663aed0de" (UID: "6862987c-7402-4134-a7a7-4a1663aed0de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.398899 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.398871 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-istio-token" pod "6862987c-7402-4134-a7a7-4a1663aed0de" (UID: "6862987c-7402-4134-a7a7-4a1663aed0de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:53.399126 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.399086 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6862987c-7402-4134-a7a7-4a1663aed0de-istio-podinfo" pod "6862987c-7402-4134-a7a7-4a1663aed0de" (UID: "6862987c-7402-4134-a7a7-4a1663aed0de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:53.399287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.399268 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-envoy" pod "6862987c-7402-4134-a7a7-4a1663aed0de" (UID: "6862987c-7402-4134-a7a7-4a1663aed0de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:53.399339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.399292 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-kube-api-access-fmzbt" pod "6862987c-7402-4134-a7a7-4a1663aed0de" (UID: "6862987c-7402-4134-a7a7-4a1663aed0de"). InnerVolumeSpecName "kube-api-access-fmzbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:53.497074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.496987 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab0a623-d106-4608-817a-6445d1d0cf39-kube-api-access-4bl9s\" (UniqueName: \"kubernetes.io/projected/1ab0a623-d106-4608-817a-6445d1d0cf39-kube-api-access-4bl9s\") pod \"1ab0a623-d106-4608-817a-6445d1d0cf39\" (UID: \"1ab0a623-d106-4608-817a-6445d1d0cf39\") " Oct 29 22:46:53.497074 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497039 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab0a623-d106-4608-817a-6445d1d0cf39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ab0a623-d106-4608-817a-6445d1d0cf39-plugins-config-volume\") pod \"1ab0a623-d106-4608-817a-6445d1d0cf39\" (UID: \"1ab0a623-d106-4608-817a-6445d1d0cf39\") " Oct 29 22:46:53.497385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497368 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmzbt\" (UniqueName: \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-kube-api-access-fmzbt\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.497464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497386 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6862987c-7402-4134-a7a7-4a1663aed0de-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.497464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497395 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.497464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497403 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.497464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497415 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.497464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497427 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.497464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497439 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6862987c-7402-4134-a7a7-4a1663aed0de-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.497464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497447 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6862987c-7402-4134-a7a7-4a1663aed0de-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.497464 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497455 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6862987c-7402-4134-a7a7-4a1663aed0de-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.497759 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.497498 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab0a623-d106-4608-817a-6445d1d0cf39-plugins-config-volume" pod "1ab0a623-d106-4608-817a-6445d1d0cf39" (UID: "1ab0a623-d106-4608-817a-6445d1d0cf39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:53.499099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.499076 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab0a623-d106-4608-817a-6445d1d0cf39-kube-api-access-4bl9s" pod "1ab0a623-d106-4608-817a-6445d1d0cf39" (UID: "1ab0a623-d106-4608-817a-6445d1d0cf39"). InnerVolumeSpecName "kube-api-access-4bl9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:53.598966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.598932 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bl9s\" (UniqueName: \"kubernetes.io/projected/1ab0a623-d106-4608-817a-6445d1d0cf39-kube-api-access-4bl9s\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.598966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.598960 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ab0a623-d106-4608-817a-6445d1d0cf39-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:46:53.656144 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.656108 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" podUID="ad3b2165-83e9-4020-8159-f045ac452bd0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:53.809288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.809195 4952 generic.go:356] "Generic (PLEG): container finished" podID="6862987c-7402-4134-a7a7-4a1663aed0de" containerID="7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b" exitCode=0 Oct 29 22:46:53.809288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.809249 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" Oct 29 22:46:53.809513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.809282 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" event={"ID":"6862987c-7402-4134-a7a7-4a1663aed0de","Type":"ContainerDied","Data":"7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b"} Oct 29 22:46:53.809513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.809348 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v" event={"ID":"6862987c-7402-4134-a7a7-4a1663aed0de","Type":"ContainerDied","Data":"b7295427a6f163f22cc89868e8b9a53905c9fdedf59c649f2386ba297c5ab93e"} Oct 29 22:46:53.809513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.809379 4952 scope.go:122] "RemoveContainer" containerID="7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b" Oct 29 22:46:53.810943 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.810898 4952 generic.go:356] "Generic (PLEG): container finished" podID="1ab0a623-d106-4608-817a-6445d1d0cf39" containerID="95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19" exitCode=0 Oct 29 22:46:53.811066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.810951 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" Oct 29 22:46:53.811066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.810997 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" event={"ID":"1ab0a623-d106-4608-817a-6445d1d0cf39","Type":"ContainerDied","Data":"95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19"} Oct 29 22:46:53.811066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.811044 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" event={"ID":"1ab0a623-d106-4608-817a-6445d1d0cf39","Type":"ContainerDied","Data":"0e81b932413a9c40274d782e35f9e1b59ba5b8595f3413c194190881d3b79bda"} Oct 29 22:46:53.822117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.821961 4952 scope.go:122] "RemoveContainer" containerID="62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39" Oct 29 22:46:53.833023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.832998 4952 scope.go:122] "RemoveContainer" containerID="7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b" Oct 29 22:46:53.833357 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:46:53.833334 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b\": container with ID starting with 7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b not found: ID does not exist" containerID="7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b" Oct 29 22:46:53.833447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.833371 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b"} err="failed to get container status \"7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b\": rpc error: code = NotFound desc = could not find container \"7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b\": container with ID starting with 7a0a933ad6607158aefe10c3daf913308199a0c7ed8498e0f32d622586f19b7b not found: ID does not exist" Oct 29 22:46:53.833447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.833396 4952 scope.go:122] "RemoveContainer" containerID="62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39" Oct 29 22:46:53.833685 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:46:53.833662 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39\": container with ID starting with 62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39 not found: ID does not exist" containerID="62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39" Oct 29 22:46:53.833728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.833693 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39"} err="failed to get container status \"62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39\": rpc error: code = NotFound desc = could not find container \"62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39\": container with ID starting with 62e517f034f2b1780b9e188d03fca0e53b51e619636ff5f874f2763269b3aa39 not found: ID does not exist" Oct 29 22:46:53.833728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.833710 4952 scope.go:122] "RemoveContainer" containerID="95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19" Oct 29 22:46:53.848366 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.848342 4952 scope.go:122] "RemoveContainer" containerID="95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19" Oct 29 22:46:53.848718 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:46:53.848687 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19\": container with ID starting with 95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19 not found: ID does not exist" containerID="95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19" Oct 29 22:46:53.848831 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.848725 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19"} err="failed to get container status \"95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19\": rpc error: code = NotFound desc = could not find container \"95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19\": container with ID starting with 95ed2ae89cd7f3c8cf84ec0b1c000fc750150de8730fea8d1e5f7c9e2a7e0d19 not found: ID does not exist" Oct 29 22:46:53.873496 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.873451 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7"] Oct 29 22:46:53.879009 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.878977 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7"] Oct 29 22:46:53.916846 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.916814 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v"] Oct 29 22:46:53.921251 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:53.921214 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-580af2-openshift-default-66c86c79fc-h999v"] Oct 29 22:46:54.282804 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:54.282756 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3b067-db9f66687-4l6j7" podUID="1ab0a623-d106-4608-817a-6445d1d0cf39" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.59:50052\" within 1s: context deadline exceeded" Oct 29 22:46:54.283005 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:46:54.282979 4952 logging.go:55] [core] [Channel #3536 SubChannel #3537]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.59:50052", ServerName: "10.132.0.59:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.59:50052: operation was canceled" Oct 29 22:46:54.638435 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:54.638381 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:54.645970 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:54.645944 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab0a623-d106-4608-817a-6445d1d0cf39" path="/var/lib/kubelet/pods/1ab0a623-d106-4608-817a-6445d1d0cf39/volumes" Oct 29 22:46:54.646445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:54.646430 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" path="/var/lib/kubelet/pods/6862987c-7402-4134-a7a7-4a1663aed0de/volumes" Oct 29 22:46:54.809558 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:54.809519 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:55.025113 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:55.025023 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:56.644083 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:56.644043 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:57.287414 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:57.287377 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:57.462056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:57.462021 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:57.558648 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:57.558559 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj"] Oct 29 22:46:57.559244 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:57.559165 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" containerID="cri-o://3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269" gracePeriod=30 Oct 29 22:46:57.657445 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:57.657404 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:58.211140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:58.211107 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:58.409805 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:58.409765 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:59.262690 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:59.262649 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:59.553955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:46:59.553845 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:00.027107 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:00.027073 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:00.701017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:00.700978 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:00.877211 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:00.877175 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:01.726144 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:01.726106 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:01.874976 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:01.874939 4952 generic.go:356] "Generic (PLEG): container finished" podID="ad3b2165-83e9-4020-8159-f045ac452bd0" containerID="fe0951340f00dea00f2dd9d59a01089c5f23c2af3936c5fb19e94f268c9369ed" exitCode=0 Oct 29 22:47:01.875157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:01.874993 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" event={"ID":"ad3b2165-83e9-4020-8159-f045ac452bd0","Type":"ContainerDied","Data":"fe0951340f00dea00f2dd9d59a01089c5f23c2af3936c5fb19e94f268c9369ed"} Oct 29 22:47:01.919032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:01.918998 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:47:02.088041 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.088007 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3b2165-83e9-4020-8159-f045ac452bd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3b2165-83e9-4020-8159-f045ac452bd0-plugins-config-volume\") pod \"ad3b2165-83e9-4020-8159-f045ac452bd0\" (UID: \"ad3b2165-83e9-4020-8159-f045ac452bd0\") " Oct 29 22:47:02.088257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.088082 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3b2165-83e9-4020-8159-f045ac452bd0-kube-api-access-ssbzr\" (UniqueName: \"kubernetes.io/projected/ad3b2165-83e9-4020-8159-f045ac452bd0-kube-api-access-ssbzr\") pod \"ad3b2165-83e9-4020-8159-f045ac452bd0\" (UID: \"ad3b2165-83e9-4020-8159-f045ac452bd0\") " Oct 29 22:47:02.088472 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.088444 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3b2165-83e9-4020-8159-f045ac452bd0-plugins-config-volume" pod "ad3b2165-83e9-4020-8159-f045ac452bd0" (UID: "ad3b2165-83e9-4020-8159-f045ac452bd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:02.090244 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.090219 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3b2165-83e9-4020-8159-f045ac452bd0-kube-api-access-ssbzr" pod "ad3b2165-83e9-4020-8159-f045ac452bd0" (UID: "ad3b2165-83e9-4020-8159-f045ac452bd0"). InnerVolumeSpecName "kube-api-access-ssbzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:02.189662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.189564 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3b2165-83e9-4020-8159-f045ac452bd0-plugins-config-volume\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.189662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.189612 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssbzr\" (UniqueName: \"kubernetes.io/projected/ad3b2165-83e9-4020-8159-f045ac452bd0-kube-api-access-ssbzr\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.777688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.777651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:02.834594 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.834572 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 22:47:02.886360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.886285 4952 generic.go:356] "Generic (PLEG): container finished" podID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerID="3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269" exitCode=0 Oct 29 22:47:02.886360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.886344 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" Oct 29 22:47:02.886568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.886370 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" event={"ID":"7cde5d96-3692-421c-81b2-44ad13d35d1e","Type":"ContainerDied","Data":"3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269"} Oct 29 22:47:02.886568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.886432 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj" event={"ID":"7cde5d96-3692-421c-81b2-44ad13d35d1e","Type":"ContainerDied","Data":"f2adb4d2a4cbc3e17624bfd6e33696636ecf9bf0aca0630d5f541bbe09c16a78"} Oct 29 22:47:02.886568 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.886464 4952 scope.go:122] "RemoveContainer" containerID="3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269" Oct 29 22:47:02.888197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.888166 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" Oct 29 22:47:02.888197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.888182 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789" event={"ID":"ad3b2165-83e9-4020-8159-f045ac452bd0","Type":"ContainerDied","Data":"2442214bcc7e1bcd84b4eef06a89873fd75ebb5f5baa7754149e898e5a2aa9cc"} Oct 29 22:47:02.894020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.893998 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-token\" (UniqueName: \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-token\") pod \"7cde5d96-3692-421c-81b2-44ad13d35d1e\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " Oct 29 22:47:02.894140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.894035 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-socket\") pod \"7cde5d96-3692-421c-81b2-44ad13d35d1e\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " Oct 29 22:47:02.894140 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.894060 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-data\") pod \"7cde5d96-3692-421c-81b2-44ad13d35d1e\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " Oct 29 22:47:02.894389 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.894365 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-socket" pod "7cde5d96-3692-421c-81b2-44ad13d35d1e" (UID: "7cde5d96-3692-421c-81b2-44ad13d35d1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:02.894511 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.894391 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-data" pod "7cde5d96-3692-421c-81b2-44ad13d35d1e" (UID: "7cde5d96-3692-421c-81b2-44ad13d35d1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:02.896207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.896178 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-token" pod "7cde5d96-3692-421c-81b2-44ad13d35d1e" (UID: "7cde5d96-3692-421c-81b2-44ad13d35d1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:02.897894 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.897866 4952 scope.go:122] "RemoveContainer" containerID="3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269" Oct 29 22:47:02.898271 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:47:02.898241 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269\": container with ID starting with 3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269 not found: ID does not exist" containerID="3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269" Oct 29 22:47:02.898408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.898282 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269"} err="failed to get container status \"3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269\": rpc error: code = NotFound desc = could not find container \"3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269\": container with ID starting with 3971a2fff828e6501e94b88c08d531c8ec51551f2f74233586ba967cd39a7269 not found: ID does not exist" Oct 29 22:47:02.898408 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.898307 4952 scope.go:122] "RemoveContainer" containerID="fe0951340f00dea00f2dd9d59a01089c5f23c2af3936c5fb19e94f268c9369ed" Oct 29 22:47:02.957772 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.957737 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789"] Oct 29 22:47:02.961355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.961329 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-580af2-6c8f6986dc-qp789"] Oct 29 22:47:02.995341 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.995306 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-kube-api-access-rkxdz\" (UniqueName: \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-kube-api-access-rkxdz\") pod \"7cde5d96-3692-421c-81b2-44ad13d35d1e\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " Oct 29 22:47:02.995505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.995358 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-credential-socket\") pod \"7cde5d96-3692-421c-81b2-44ad13d35d1e\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " Oct 29 22:47:02.995505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.995396 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cde5d96-3692-421c-81b2-44ad13d35d1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cde5d96-3692-421c-81b2-44ad13d35d1e-istiod-ca-cert\") pod \"7cde5d96-3692-421c-81b2-44ad13d35d1e\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " Oct 29 22:47:02.995505 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.995427 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-podinfo\") pod \"7cde5d96-3692-421c-81b2-44ad13d35d1e\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " Oct 29 22:47:02.995689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.995513 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-certs\") pod \"7cde5d96-3692-421c-81b2-44ad13d35d1e\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " Oct 29 22:47:02.995689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.995613 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-envoy\") pod \"7cde5d96-3692-421c-81b2-44ad13d35d1e\" (UID: \"7cde5d96-3692-421c-81b2-44ad13d35d1e\") " Oct 29 22:47:02.995811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.995724 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-credential-socket" pod "7cde5d96-3692-421c-81b2-44ad13d35d1e" (UID: "7cde5d96-3692-421c-81b2-44ad13d35d1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:02.995918 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.995866 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cde5d96-3692-421c-81b2-44ad13d35d1e-istiod-ca-cert" pod "7cde5d96-3692-421c-81b2-44ad13d35d1e" (UID: "7cde5d96-3692-421c-81b2-44ad13d35d1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:02.996147 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.996120 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.996276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.996148 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cde5d96-3692-421c-81b2-44ad13d35d1e-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.996276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.996164 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.996276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.996161 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-certs" pod "7cde5d96-3692-421c-81b2-44ad13d35d1e" (UID: "7cde5d96-3692-421c-81b2-44ad13d35d1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:02.996276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.996175 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.996276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.996214 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:02.997718 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.997693 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-podinfo" pod "7cde5d96-3692-421c-81b2-44ad13d35d1e" (UID: "7cde5d96-3692-421c-81b2-44ad13d35d1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:47:02.997794 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.997753 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-envoy" pod "7cde5d96-3692-421c-81b2-44ad13d35d1e" (UID: "7cde5d96-3692-421c-81b2-44ad13d35d1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:02.997794 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:02.997787 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-kube-api-access-rkxdz" pod "7cde5d96-3692-421c-81b2-44ad13d35d1e" (UID: "7cde5d96-3692-421c-81b2-44ad13d35d1e"). InnerVolumeSpecName "kube-api-access-rkxdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:03.097352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:03.097316 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkxdz\" (UniqueName: \"kubernetes.io/projected/7cde5d96-3692-421c-81b2-44ad13d35d1e-kube-api-access-rkxdz\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:03.097352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:03.097348 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:03.097352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:03.097358 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:03.097582 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:03.097366 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cde5d96-3692-421c-81b2-44ad13d35d1e-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:03.238426 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:03.238393 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj"] Oct 29 22:47:03.251478 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:03.251447 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3b067-openshift-default-5974979457-f5lqj"] Oct 29 22:47:03.413816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:03.413775 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:04.667203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:04.665356 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" path="/var/lib/kubelet/pods/7cde5d96-3692-421c-81b2-44ad13d35d1e/volumes" Oct 29 22:47:04.667203 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:04.666058 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3b2165-83e9-4020-8159-f045ac452bd0" path="/var/lib/kubelet/pods/ad3b2165-83e9-4020-8159-f045ac452bd0/volumes" Oct 29 22:47:04.742091 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:04.742053 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:05.090337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:05.090292 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:05.274396 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:05.274359 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:05.539287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:05.539212 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:05.719351 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:05.719309 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:05.894989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:05.894932 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:06.391221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:06.391182 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:06.486740 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:06.486703 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:47:07.520410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:07.520313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:09.121246 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:09.121211 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:09.822013 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:09.821976 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:10.718327 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:10.718292 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:12.447655 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:12.447610 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:14.001657 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:14.001618 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:14.179468 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:14.179422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:14.351591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:14.351552 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:14.520184 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:14.520151 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:14.690859 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:14.690765 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:15.078703 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:15.078660 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:15.529162 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:47:15.529075 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:47:16.019581 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:16.019543 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:16.328001 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:16.327969 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:47:16.344529 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:16.344501 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:47:17.864676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:17.864642 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:18.395638 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:18.395597 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:47:19.561591 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:19.561555 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:19.997636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:19.997605 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw"] Oct 29 22:47:20.138224 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:20.138181 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:21.220368 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:21.220331 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:22.825663 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:22.825620 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:24.416701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:24.416656 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:25.606117 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:25.606077 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:25.787069 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:25.787028 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:25.966383 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:25.966262 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:26.152380 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:26.152340 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:26.447811 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:26.447766 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:27.063346 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:27.063301 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:27.266540 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:27.266498 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:27.741500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:27.741459 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:28.309816 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:28.309780 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:29.919427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:29.919380 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:31.291302 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:31.291269 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:47:31.728714 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:31.728675 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:33.323926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:33.323865 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:34.333040 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:34.333009 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:34.513554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:34.513505 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:34.716878 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:34.716783 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:34.930839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:34.930807 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:35.019761 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.019649 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" containerID="cri-o://730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e" gracePeriod=45 Oct 29 22:47:35.291861 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.291834 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:47:35.449502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.448347 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f7e2897-30ea-4b2e-a06d-b64329faff99-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7e2897-30ea-4b2e-a06d-b64329faff99-tls-certs\") pod \"6f7e2897-30ea-4b2e-a06d-b64329faff99\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " Oct 29 22:47:35.449502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.448441 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-tmp-dir\") pod \"6f7e2897-30ea-4b2e-a06d-b64329faff99\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " Oct 29 22:47:35.449502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.448494 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-model-cache\") pod \"6f7e2897-30ea-4b2e-a06d-b64329faff99\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " Oct 29 22:47:35.449502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.448594 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-dshm\") pod \"6f7e2897-30ea-4b2e-a06d-b64329faff99\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " Oct 29 22:47:35.449502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.448663 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-home\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-home\") pod \"6f7e2897-30ea-4b2e-a06d-b64329faff99\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " Oct 29 22:47:35.449502 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.448732 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7e2897-30ea-4b2e-a06d-b64329faff99-kube-api-access-smhrk\" (UniqueName: \"kubernetes.io/projected/6f7e2897-30ea-4b2e-a06d-b64329faff99-kube-api-access-smhrk\") pod \"6f7e2897-30ea-4b2e-a06d-b64329faff99\" (UID: \"6f7e2897-30ea-4b2e-a06d-b64329faff99\") " Oct 29 22:47:35.450319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.450125 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-tmp-dir" pod "6f7e2897-30ea-4b2e-a06d-b64329faff99" (UID: "6f7e2897-30ea-4b2e-a06d-b64329faff99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.450384 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.450339 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-model-cache" pod "6f7e2897-30ea-4b2e-a06d-b64329faff99" (UID: "6f7e2897-30ea-4b2e-a06d-b64329faff99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.457835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.457795 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7e2897-30ea-4b2e-a06d-b64329faff99-kube-api-access-smhrk" pod "6f7e2897-30ea-4b2e-a06d-b64329faff99" (UID: "6f7e2897-30ea-4b2e-a06d-b64329faff99"). InnerVolumeSpecName "kube-api-access-smhrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:35.459701 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.459665 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-home" pod "6f7e2897-30ea-4b2e-a06d-b64329faff99" (UID: "6f7e2897-30ea-4b2e-a06d-b64329faff99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.460264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.460230 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-dshm" pod "6f7e2897-30ea-4b2e-a06d-b64329faff99" (UID: "6f7e2897-30ea-4b2e-a06d-b64329faff99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:35.460893 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.460860 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7e2897-30ea-4b2e-a06d-b64329faff99-tls-certs" pod "6f7e2897-30ea-4b2e-a06d-b64329faff99" (UID: "6f7e2897-30ea-4b2e-a06d-b64329faff99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:35.551102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.551011 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.551102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.551057 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smhrk\" (UniqueName: \"kubernetes.io/projected/6f7e2897-30ea-4b2e-a06d-b64329faff99-kube-api-access-smhrk\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.551102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.551073 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7e2897-30ea-4b2e-a06d-b64329faff99-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.551102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.551087 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.551102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.551101 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:35.551415 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:35.551113 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7e2897-30ea-4b2e-a06d-b64329faff99-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:36.101153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.101115 4952 generic.go:356] "Generic (PLEG): container finished" podID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerID="730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e" exitCode=0 Oct 29 22:47:36.101345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.101179 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" Oct 29 22:47:36.101345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.101200 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" event={"ID":"6f7e2897-30ea-4b2e-a06d-b64329faff99","Type":"ContainerDied","Data":"730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e"} Oct 29 22:47:36.101345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.101245 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw" event={"ID":"6f7e2897-30ea-4b2e-a06d-b64329faff99","Type":"ContainerDied","Data":"ebc47dfdcb7fed82dd3321809a77d74f31990120b99660d184e7ae0cddc174c1"} Oct 29 22:47:36.101345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.101262 4952 scope.go:122] "RemoveContainer" containerID="730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e" Oct 29 22:47:36.112197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.112179 4952 scope.go:122] "RemoveContainer" containerID="730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e" Oct 29 22:47:36.112454 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:47:36.112431 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e\": container with ID starting with 730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e not found: ID does not exist" containerID="730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e" Oct 29 22:47:36.112516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.112461 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e"} err="failed to get container status \"730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e\": rpc error: code = NotFound desc = could not find container \"730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e\": container with ID starting with 730f308c5d9e5cfc797783fad89e3dd4fe6e5652c1c315ab2ab6afc4b0f9323e not found: ID does not exist" Oct 29 22:47:36.162056 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.162018 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw"] Oct 29 22:47:36.175438 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.175402 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-863f1b/simulator-w6-863f1b-kserve-57dd578dfc-bmcfw"] Oct 29 22:47:36.645401 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.645354 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" path="/var/lib/kubelet/pods/6f7e2897-30ea-4b2e-a06d-b64329faff99/volumes" Oct 29 22:47:36.804496 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:36.804461 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:38.486953 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:38.486887 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:38.680452 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:38.680415 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:38.910334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:38.910296 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:39.155687 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:39.155643 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:39.407940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:39.407870 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:40.282337 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:40.282297 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:42.049369 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:42.049329 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:42.599646 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:42.599609 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:43.745261 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:43.745211 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:45.298101 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:45.298062 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:45.526965 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:47:45.526894 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:47:45.975023 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:45.974979 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:46.219754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:46.219708 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:46.873420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:46.873357 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:48.502836 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:48.502793 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:49.128184 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:49.128145 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:49.292989 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:49.292947 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:47:49.300352 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:49.300316 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:49.739287 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:49.739247 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm"] Oct 29 22:47:49.739801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:49.739505 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" podUID="4e22089b-9ec5-4f04-98c7-8e3a373f2dc5" containerName="maas-api" containerID="cri-o://c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a" gracePeriod=30 Oct 29 22:47:49.862032 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:49.861998 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:50.047688 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.047659 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 22:47:50.112882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.109383 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-kube-api-access-mzncp\" (UniqueName: \"kubernetes.io/projected/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-kube-api-access-mzncp\") pod \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " Oct 29 22:47:50.112882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.109469 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-metrics-certs\") pod \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " Oct 29 22:47:50.112882 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.109521 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-maas-api-tls\") pod \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " Oct 29 22:47:50.119318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.119241 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-maas-api-tls" pod "4e22089b-9ec5-4f04-98c7-8e3a373f2dc5" (UID: "4e22089b-9ec5-4f04-98c7-8e3a373f2dc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:50.119770 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.119716 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-kube-api-access-mzncp" pod "4e22089b-9ec5-4f04-98c7-8e3a373f2dc5" (UID: "4e22089b-9ec5-4f04-98c7-8e3a373f2dc5"). InnerVolumeSpecName "kube-api-access-mzncp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:50.121089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.121057 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-metrics-certs" pod "4e22089b-9ec5-4f04-98c7-8e3a373f2dc5" (UID: "4e22089b-9ec5-4f04-98c7-8e3a373f2dc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:47:50.205396 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.205354 4952 generic.go:356] "Generic (PLEG): container finished" podID="4e22089b-9ec5-4f04-98c7-8e3a373f2dc5" containerID="c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a" exitCode=0 Oct 29 22:47:50.205609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.205471 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" Oct 29 22:47:50.205609 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.205505 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" event={"ID":"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5","Type":"ContainerDied","Data":"c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a"} Oct 29 22:47:50.205732 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.205606 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm" event={"ID":"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5","Type":"ContainerDied","Data":"e13f399ac08b49ae6bfb45af420037a14ea3fae4e6bc64f1f49fed8691e7acc7"} Oct 29 22:47:50.205732 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.205641 4952 scope.go:122] "RemoveContainer" containerID="c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a" Oct 29 22:47:50.210197 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.210175 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-openshift-service-ca\") pod \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\" (UID: \"4e22089b-9ec5-4f04-98c7-8e3a373f2dc5\") " Oct 29 22:47:50.210451 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.210436 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:50.210537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.210455 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzncp\" (UniqueName: \"kubernetes.io/projected/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-kube-api-access-mzncp\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:50.210537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.210470 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:50.210537 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.210511 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-openshift-service-ca" pod "4e22089b-9ec5-4f04-98c7-8e3a373f2dc5" (UID: "4e22089b-9ec5-4f04-98c7-8e3a373f2dc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:50.216232 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.216199 4952 scope.go:122] "RemoveContainer" containerID="c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a" Oct 29 22:47:50.216541 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:47:50.216514 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a\": container with ID starting with c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a not found: ID does not exist" containerID="c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a" Oct 29 22:47:50.216616 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.216548 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a"} err="failed to get container status \"c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a\": rpc error: code = NotFound desc = could not find container \"c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a\": container with ID starting with c471c5b250e683b5251b54673f13572a2be52dfce4cf2ce0bc90ae28fc81db2a not found: ID does not exist" Oct 29 22:47:50.311599 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.311502 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:47:50.384444 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.384410 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:50.557512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.557472 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm"] Oct 29 22:47:50.559547 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.559509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:50.563363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.563301 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-863f1b-7bbb478469-6msjm"] Oct 29 22:47:50.643363 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:50.643320 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e22089b-9ec5-4f04-98c7-8e3a373f2dc5" path="/var/lib/kubelet/pods/4e22089b-9ec5-4f04-98c7-8e3a373f2dc5/volumes" Oct 29 22:47:51.053410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:51.053377 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:51.895544 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:51.895504 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:52.327152 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:52.327117 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:52.497693 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:52.497653 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:52.688807 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:52.688717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:53.017962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:53.017874 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:54.231157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:54.231111 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:55.839027 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:55.838988 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:57.530699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:57.530657 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:58.089318 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:58.089279 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:58.275520 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:58.275477 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:58.330941 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:58.330861 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45"] Oct 29 22:47:58.331217 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:58.331173 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" containerID="cri-o://d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7" gracePeriod=30 Oct 29 22:47:58.455562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:58.455450 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:58.742201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:58.742117 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:47:58.742201 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:58.742180 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:47:59.234428 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:47:59.234390 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:01.104045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:01.104007 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:01.175266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:01.175226 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:48:01.301319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:01.301266 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:01.705109 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:01.705066 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:02.946397 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:02.946343 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:03.598326 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.598291 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:48:03.643790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.643683 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-token\" (UniqueName: \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-token\") pod \"7dc40683-1682-4064-a856-b7c8dd137a0a\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " Oct 29 22:48:03.643790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.643747 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-envoy\") pod \"7dc40683-1682-4064-a856-b7c8dd137a0a\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " Oct 29 22:48:03.643790 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.643783 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-podinfo\") pod \"7dc40683-1682-4064-a856-b7c8dd137a0a\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " Oct 29 22:48:03.644214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.643810 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-kube-api-access-c4264\" (UniqueName: \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-kube-api-access-c4264\") pod \"7dc40683-1682-4064-a856-b7c8dd137a0a\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " Oct 29 22:48:03.644214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.643891 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc40683-1682-4064-a856-b7c8dd137a0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc40683-1682-4064-a856-b7c8dd137a0a-istiod-ca-cert\") pod \"7dc40683-1682-4064-a856-b7c8dd137a0a\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " Oct 29 22:48:03.644214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.643952 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-socket\") pod \"7dc40683-1682-4064-a856-b7c8dd137a0a\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " Oct 29 22:48:03.644214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.644025 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-credential-socket\") pod \"7dc40683-1682-4064-a856-b7c8dd137a0a\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " Oct 29 22:48:03.644214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.644082 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-certs\") pod \"7dc40683-1682-4064-a856-b7c8dd137a0a\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " Oct 29 22:48:03.644214 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.644123 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-data\") pod \"7dc40683-1682-4064-a856-b7c8dd137a0a\" (UID: \"7dc40683-1682-4064-a856-b7c8dd137a0a\") " Oct 29 22:48:03.644705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.644580 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc40683-1682-4064-a856-b7c8dd137a0a-istiod-ca-cert" pod "7dc40683-1682-4064-a856-b7c8dd137a0a" (UID: "7dc40683-1682-4064-a856-b7c8dd137a0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:03.644705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.644656 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-credential-socket" pod "7dc40683-1682-4064-a856-b7c8dd137a0a" (UID: "7dc40683-1682-4064-a856-b7c8dd137a0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:03.645230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.644895 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-certs" pod "7dc40683-1682-4064-a856-b7c8dd137a0a" (UID: "7dc40683-1682-4064-a856-b7c8dd137a0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:03.645230 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.645017 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-data" pod "7dc40683-1682-4064-a856-b7c8dd137a0a" (UID: "7dc40683-1682-4064-a856-b7c8dd137a0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:03.645354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.645291 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.645354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.645316 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc40683-1682-4064-a856-b7c8dd137a0a-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.645354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.645331 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.645354 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.645344 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.645666 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.645639 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-socket" pod "7dc40683-1682-4064-a856-b7c8dd137a0a" (UID: "7dc40683-1682-4064-a856-b7c8dd137a0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:03.646603 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.646581 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-token" pod "7dc40683-1682-4064-a856-b7c8dd137a0a" (UID: "7dc40683-1682-4064-a856-b7c8dd137a0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:03.647774 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.647756 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-envoy" pod "7dc40683-1682-4064-a856-b7c8dd137a0a" (UID: "7dc40683-1682-4064-a856-b7c8dd137a0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:03.647868 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.647838 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-kube-api-access-c4264" pod "7dc40683-1682-4064-a856-b7c8dd137a0a" (UID: "7dc40683-1682-4064-a856-b7c8dd137a0a"). InnerVolumeSpecName "kube-api-access-c4264". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:03.647958 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.647862 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-podinfo" pod "7dc40683-1682-4064-a856-b7c8dd137a0a" (UID: "7dc40683-1682-4064-a856-b7c8dd137a0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:48:03.746636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.746594 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.746636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.746632 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.746636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.746644 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.746999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.746652 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc40683-1682-4064-a856-b7c8dd137a0a-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:03.746999 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:03.746660 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4264\" (UniqueName: \"kubernetes.io/projected/7dc40683-1682-4064-a856-b7c8dd137a0a-kube-api-access-c4264\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:04.308695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.308656 4952 generic.go:356] "Generic (PLEG): container finished" podID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerID="d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7" exitCode=0 Oct 29 22:48:04.309210 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.308708 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" Oct 29 22:48:04.309210 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.308746 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" event={"ID":"7dc40683-1682-4064-a856-b7c8dd137a0a","Type":"ContainerDied","Data":"d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7"} Oct 29 22:48:04.309210 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.308802 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45" event={"ID":"7dc40683-1682-4064-a856-b7c8dd137a0a","Type":"ContainerDied","Data":"c542b508fa463a18896ee24ff5b91451f2924c233f984dbf68b36f0a3633de0c"} Oct 29 22:48:04.309210 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.308828 4952 scope.go:122] "RemoveContainer" containerID="d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7" Oct 29 22:48:04.329089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.321143 4952 scope.go:122] "RemoveContainer" containerID="d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7" Oct 29 22:48:04.329089 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:48:04.321592 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7\": container with ID starting with d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7 not found: ID does not exist" containerID="d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7" Oct 29 22:48:04.329089 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.321634 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7"} err="failed to get container status \"d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7\": rpc error: code = NotFound desc = could not find container \"d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7\": container with ID starting with d5cdbf00a4d40d065f3f8636a17238550616521005e428b62fc7d216896009c7 not found: ID does not exist" Oct 29 22:48:04.381257 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.381217 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45"] Oct 29 22:48:04.394266 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.394233 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-863f1b-openshift-default-6b4b8c98cc-jrl45"] Oct 29 22:48:04.642830 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.642788 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:04.644956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:04.644920 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" path="/var/lib/kubelet/pods/7dc40683-1682-4064-a856-b7c8dd137a0a/volumes" Oct 29 22:48:06.258207 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:06.258171 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:07.900895 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:07.900862 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:09.513345 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:09.513313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:11.029025 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:11.028985 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:12.714765 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:12.714727 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:12.900717 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:12.900679 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:13.096116 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:13.096079 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:13.917852 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:13.917813 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2"] Oct 29 22:48:13.918384 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:13.918107 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" podUID="b89a0aba-122c-49b4-95c5-c38bd87cbda9" containerName="maas-api" containerID="cri-o://b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69" gracePeriod=30 Oct 29 22:48:14.037669 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.037633 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:14.199385 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.199356 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:48:14.365894 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.365862 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-metrics-certs\") pod \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " Oct 29 22:48:14.366102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.365967 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b89a0aba-122c-49b4-95c5-c38bd87cbda9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89a0aba-122c-49b4-95c5-c38bd87cbda9-openshift-service-ca\") pod \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " Oct 29 22:48:14.366102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.366056 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-maas-api-tls\") pod \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " Oct 29 22:48:14.366102 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.366082 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89a0aba-122c-49b4-95c5-c38bd87cbda9-kube-api-access-q8brl\" (UniqueName: \"kubernetes.io/projected/b89a0aba-122c-49b4-95c5-c38bd87cbda9-kube-api-access-q8brl\") pod \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\" (UID: \"b89a0aba-122c-49b4-95c5-c38bd87cbda9\") " Oct 29 22:48:14.366387 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.366363 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89a0aba-122c-49b4-95c5-c38bd87cbda9-openshift-service-ca" pod "b89a0aba-122c-49b4-95c5-c38bd87cbda9" (UID: "b89a0aba-122c-49b4-95c5-c38bd87cbda9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:14.368221 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.368192 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89a0aba-122c-49b4-95c5-c38bd87cbda9-kube-api-access-q8brl" pod "b89a0aba-122c-49b4-95c5-c38bd87cbda9" (UID: "b89a0aba-122c-49b4-95c5-c38bd87cbda9"). InnerVolumeSpecName "kube-api-access-q8brl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:14.368332 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.368273 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-maas-api-tls" pod "b89a0aba-122c-49b4-95c5-c38bd87cbda9" (UID: "b89a0aba-122c-49b4-95c5-c38bd87cbda9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:14.368510 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.368488 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-metrics-certs" pod "b89a0aba-122c-49b4-95c5-c38bd87cbda9" (UID: "b89a0aba-122c-49b4-95c5-c38bd87cbda9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:14.374253 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.374230 4952 generic.go:356] "Generic (PLEG): container finished" podID="b89a0aba-122c-49b4-95c5-c38bd87cbda9" containerID="b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69" exitCode=0 Oct 29 22:48:14.374358 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.374296 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" Oct 29 22:48:14.374412 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.374296 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" event={"ID":"b89a0aba-122c-49b4-95c5-c38bd87cbda9","Type":"ContainerDied","Data":"b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69"} Oct 29 22:48:14.374455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.374420 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2" event={"ID":"b89a0aba-122c-49b4-95c5-c38bd87cbda9","Type":"ContainerDied","Data":"2be8f40931d7f77317f96925045faa702235a9b3a2071eed920ef9da1d69d843"} Oct 29 22:48:14.374455 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.374443 4952 scope.go:122] "RemoveContainer" containerID="b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69" Oct 29 22:48:14.393761 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.393742 4952 scope.go:122] "RemoveContainer" containerID="b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69" Oct 29 22:48:14.394111 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:48:14.394089 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69\": container with ID starting with b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69 not found: ID does not exist" containerID="b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69" Oct 29 22:48:14.394190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.394119 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69"} err="failed to get container status \"b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69\": rpc error: code = NotFound desc = could not find container \"b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69\": container with ID starting with b6bb00d6b8ef0605aead5e4ed5a63286923926301cb9afe6388570496c51ec69 not found: ID does not exist" Oct 29 22:48:14.432604 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.432528 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2"] Oct 29 22:48:14.438123 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.438099 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8d569e-db64bdf79-2bzs2"] Oct 29 22:48:14.443131 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.443098 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:14.467262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.467237 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89a0aba-122c-49b4-95c5-c38bd87cbda9-openshift-service-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:14.467262 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.467259 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-maas-api-tls\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:14.467421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.467270 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8brl\" (UniqueName: \"kubernetes.io/projected/b89a0aba-122c-49b4-95c5-c38bd87cbda9-kube-api-access-q8brl\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:14.467421 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.467280 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b89a0aba-122c-49b4-95c5-c38bd87cbda9-metrics-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:14.614747 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.614696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:14.644621 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.644584 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b89a0aba-122c-49b4-95c5-c38bd87cbda9" path="/var/lib/kubelet/pods/b89a0aba-122c-49b4-95c5-c38bd87cbda9/volumes" Oct 29 22:48:14.795517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:14.795412 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:15.184342 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:15.184301 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:15.527675 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:48:15.527588 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:48:16.328120 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:16.328082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:17.205305 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:17.205259 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:48:18.017545 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:18.017510 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:19.635689 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:19.635651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:21.226619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:21.226578 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:21.974018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:21.973982 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:22.148898 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:22.148861 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:22.234569 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:22.234477 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc"] Oct 29 22:48:22.235012 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:22.234704 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" containerID="cri-o://e92f37f886a8623f2a4151fc151cecb529978a1cf773b2f762481247ee783e43" gracePeriod=30 Oct 29 22:48:22.338190 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:22.338151 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:22.837400 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:22.837358 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:23.011871 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:23.011821 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:24.750461 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:24.750417 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:26.314340 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:26.314304 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:27.457126 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.457095 4952 generic.go:356] "Generic (PLEG): container finished" podID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerID="e92f37f886a8623f2a4151fc151cecb529978a1cf773b2f762481247ee783e43" exitCode=0 Oct 29 22:48:27.457605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.457177 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" event={"ID":"345f5596-f75f-4aa1-adc4-d97f16ffa1a0","Type":"ContainerDied","Data":"e92f37f886a8623f2a4151fc151cecb529978a1cf773b2f762481247ee783e43"} Oct 29 22:48:27.457605 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.457260 4952 scope.go:122] "RemoveContainer" containerID="c36746517a2d58b39e1afad171c5503974227c8c7aefe38d004de031be7e156c" Oct 29 22:48:27.506516 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.506458 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:48:27.587622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.587589 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-podinfo\") pod \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " Oct 29 22:48:27.587837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.587630 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-token\" (UniqueName: \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-token\") pod \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " Oct 29 22:48:27.587837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.587667 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-socket\") pod \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " Oct 29 22:48:27.587837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.587691 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-credential-socket\") pod \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " Oct 29 22:48:27.587837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.587717 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-envoy\") pod \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " Oct 29 22:48:27.587837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.587734 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istiod-ca-cert\") pod \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " Oct 29 22:48:27.587837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.587793 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-certs\") pod \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " Oct 29 22:48:27.587837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.587827 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-kube-api-access-tkwbq\" (UniqueName: \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-kube-api-access-tkwbq\") pod \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " Oct 29 22:48:27.588272 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.587885 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-data\") pod \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\" (UID: \"345f5596-f75f-4aa1-adc4-d97f16ffa1a0\") " Oct 29 22:48:27.588498 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.588466 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-socket" pod "345f5596-f75f-4aa1-adc4-d97f16ffa1a0" (UID: "345f5596-f75f-4aa1-adc4-d97f16ffa1a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:27.588891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.588845 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-credential-socket" pod "345f5596-f75f-4aa1-adc4-d97f16ffa1a0" (UID: "345f5596-f75f-4aa1-adc4-d97f16ffa1a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:27.589017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.588863 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-data" pod "345f5596-f75f-4aa1-adc4-d97f16ffa1a0" (UID: "345f5596-f75f-4aa1-adc4-d97f16ffa1a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:27.589017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.588923 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istiod-ca-cert" pod "345f5596-f75f-4aa1-adc4-d97f16ffa1a0" (UID: "345f5596-f75f-4aa1-adc4-d97f16ffa1a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:27.589017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.588943 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-certs" pod "345f5596-f75f-4aa1-adc4-d97f16ffa1a0" (UID: "345f5596-f75f-4aa1-adc4-d97f16ffa1a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:27.590293 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.590266 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-token" pod "345f5596-f75f-4aa1-adc4-d97f16ffa1a0" (UID: "345f5596-f75f-4aa1-adc4-d97f16ffa1a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:27.590543 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.590517 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-podinfo" pod "345f5596-f75f-4aa1-adc4-d97f16ffa1a0" (UID: "345f5596-f75f-4aa1-adc4-d97f16ffa1a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:48:27.590783 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.590759 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-kube-api-access-tkwbq" pod "345f5596-f75f-4aa1-adc4-d97f16ffa1a0" (UID: "345f5596-f75f-4aa1-adc4-d97f16ffa1a0"). InnerVolumeSpecName "kube-api-access-tkwbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:27.590992 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.590972 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-envoy" pod "345f5596-f75f-4aa1-adc4-d97f16ffa1a0" (UID: "345f5596-f75f-4aa1-adc4-d97f16ffa1a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:27.689484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.689454 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkwbq\" (UniqueName: \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-kube-api-access-tkwbq\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.689484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.689481 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-data\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.689484 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.689491 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-podinfo\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.689705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.689499 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-token\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.689705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.689508 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.689705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.689517 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-credential-socket\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.689705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.689524 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istio-envoy\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.689705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.689532 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-istiod-ca-cert\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.689705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.689540 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345f5596-f75f-4aa1-adc4-d97f16ffa1a0-workload-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.757295 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.757211 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5"] Oct 29 22:48:27.766990 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.766959 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq"] Oct 29 22:48:27.922962 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:27.922888 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:28.097267 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:28.097230 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:28.464288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:28.464207 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" Oct 29 22:48:28.464288 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:28.464249 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc" event={"ID":"345f5596-f75f-4aa1-adc4-d97f16ffa1a0","Type":"ContainerDied","Data":"04ebe3583d11551286f3cbc9eaa0e1276c5a4af208d68728160b0dcce07350ba"} Oct 29 22:48:28.464736 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:28.464314 4952 scope.go:122] "RemoveContainer" containerID="e92f37f886a8623f2a4151fc151cecb529978a1cf773b2f762481247ee783e43" Oct 29 22:48:28.535497 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:28.535462 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc"] Oct 29 22:48:28.548470 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:28.548441 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8d569e-openshift-default-666576f59c-zc7jc"] Oct 29 22:48:28.642728 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:28.642695 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" path="/var/lib/kubelet/pods/345f5596-f75f-4aa1-adc4-d97f16ffa1a0/volumes" Oct 29 22:48:29.518881 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:29.518845 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:31.111208 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:31.111168 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:32.720705 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:32.720673 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:33.683319 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:33.683283 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:33.850874 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:33.850834 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:34.324542 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:34.324506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:35.587565 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.587526 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d"] Oct 29 22:48:35.587991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.587967 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" Oct 29 22:48:35.587991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.587983 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" Oct 29 22:48:35.588071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588002 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4e22089b-9ec5-4f04-98c7-8e3a373f2dc5" containerName="maas-api" Oct 29 22:48:35.588071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588011 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e22089b-9ec5-4f04-98c7-8e3a373f2dc5" containerName="maas-api" Oct 29 22:48:35.588071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588027 4952 cpu_manager.go:414] "containerMap: removing container" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" Oct 29 22:48:35.588071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588034 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" Oct 29 22:48:35.588071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588046 4952 cpu_manager.go:414] "containerMap: removing container" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" Oct 29 22:48:35.588071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588051 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" Oct 29 22:48:35.588071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588058 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" Oct 29 22:48:35.588071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588062 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" Oct 29 22:48:35.588071 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588070 4952 cpu_manager.go:414] "containerMap: removing container" podUID="1ab0a623-d106-4608-817a-6445d1d0cf39" containerName="payload-pre-processing" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588075 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab0a623-d106-4608-817a-6445d1d0cf39" containerName="payload-pre-processing" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588087 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b89a0aba-122c-49b4-95c5-c38bd87cbda9" containerName="maas-api" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588092 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89a0aba-122c-49b4-95c5-c38bd87cbda9" containerName="maas-api" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588101 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6e76b81a-f826-46f6-95c7-27ed04ded082" containerName="payload-processing" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588106 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e76b81a-f826-46f6-95c7-27ed04ded082" containerName="payload-processing" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588115 4952 cpu_manager.go:414] "containerMap: removing container" podUID="bb33613f-1bfc-487d-b68c-a7f1592d2b6f" containerName="revoke-keys" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588119 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb33613f-1bfc-487d-b68c-a7f1592d2b6f" containerName="revoke-keys" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588126 4952 cpu_manager.go:414] "containerMap: removing container" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588130 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588138 4952 cpu_manager.go:414] "containerMap: removing container" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerName="cleanup" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588143 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerName="cleanup" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588149 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerName="cleanup" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588153 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerName="cleanup" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588161 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588165 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588170 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerName="cleanup" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588175 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerName="cleanup" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588181 4952 cpu_manager.go:414] "containerMap: removing container" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588185 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588193 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerName="cleanup" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588197 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fec9dea-1cd0-41f4-93c1-f19617b84ca1" containerName="cleanup" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588204 4952 cpu_manager.go:414] "containerMap: removing container" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588208 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588216 4952 cpu_manager.go:414] "containerMap: removing container" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588221 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588228 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ad3b2165-83e9-4020-8159-f045ac452bd0" containerName="payload-pre-processing" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588232 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3b2165-83e9-4020-8159-f045ac452bd0" containerName="payload-pre-processing" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588243 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588248 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588255 4952 cpu_manager.go:414] "containerMap: removing container" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerName="cleanup" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588260 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerName="cleanup" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588267 4952 cpu_manager.go:414] "containerMap: removing container" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588271 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588278 4952 cpu_manager.go:414] "containerMap: removing container" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588283 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" Oct 29 22:48:35.588355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588368 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc40683-1682-4064-a856-b7c8dd137a0a" containerName="istio-proxy" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588377 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588383 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab0a623-d106-4608-817a-6445d1d0cf39" containerName="payload-pre-processing" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588390 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89a0aba-122c-49b4-95c5-c38bd87cbda9" containerName="maas-api" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588397 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01ec1be-c7c4-4c50-bcb5-67e4b33ab788" containerName="cleanup" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588403 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e76b81a-f826-46f6-95c7-27ed04ded082" containerName="payload-processing" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588409 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb33613f-1bfc-487d-b68c-a7f1592d2b6f" containerName="revoke-keys" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588415 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cde5d96-3692-421c-81b2-44ad13d35d1e" containerName="istio-proxy" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588421 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7e2897-30ea-4b2e-a06d-b64329faff99" containerName="main" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588426 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a3f8a0-1b4d-41d1-8d9b-f0e3aac6c134" containerName="main" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588431 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588436 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588441 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3b2165-83e9-4020-8159-f045ac452bd0" containerName="payload-pre-processing" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588447 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="037a603f-52cf-4222-a5be-57f8cb73dc98" containerName="cleanup" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588452 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce3d7ba-803e-45c6-a601-33268ea9223a" containerName="cleanup" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588458 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc63e06-4fc4-4f20-ad12-1e3419d39a1c" containerName="cleanup" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588462 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ae5e2e-4abe-4ac4-8a68-0773b4098020" containerName="cleanup" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588468 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ccaa807-dd67-431a-8b48-07cf649eefed" containerName="istio-proxy" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588474 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da6a393-1fd5-4ab4-b273-c6d32454aa0c" containerName="main" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588479 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="713cded9-f796-464a-8fed-59dfbcedb3d7" containerName="cleanup" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588484 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a24bdb-4f05-4544-bd37-1896c18dffa9" containerName="cleanup" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588490 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e22089b-9ec5-4f04-98c7-8e3a373f2dc5" containerName="maas-api" Oct 29 22:48:35.589683 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.588495 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d2c654-5589-4747-986c-94ec4fd16ffd" containerName="istio-proxy" Oct 29 22:48:35.592619 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.592599 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:35.613017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.612972 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d"] Oct 29 22:48:35.622259 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.622231 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:48:35.622450 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.622248 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:48:35.622546 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.622259 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qv7bb\"" Oct 29 22:48:35.622622 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.622334 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:48:35.662018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.661965 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6777139-s2g2d\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:35.662018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.662019 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9x2\" (UniqueName: \"kubernetes.io/projected/35789086-fdba-4f26-9d20-68b3b6acb945-kube-api-access-ds9x2\") pod \"maas-api-revoke-sub-c6777139-s2g2d\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:35.662277 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.662124 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6777139-s2g2d\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:35.762994 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.762956 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6777139-s2g2d\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:35.763177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.762995 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9x2\" (UniqueName: \"kubernetes.io/projected/35789086-fdba-4f26-9d20-68b3b6acb945-kube-api-access-ds9x2\") pod \"maas-api-revoke-sub-c6777139-s2g2d\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:35.763177 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.763036 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6777139-s2g2d\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:35.763801 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.763772 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6777139-s2g2d\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:35.765528 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.765502 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6777139-s2g2d\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:35.789108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.789082 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9x2\" (UniqueName: \"kubernetes.io/projected/35789086-fdba-4f26-9d20-68b3b6acb945-kube-api-access-ds9x2\") pod \"maas-api-revoke-sub-c6777139-s2g2d\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:35.906157 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:35.906082 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:36.209775 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:36.209743 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d"] Oct 29 22:48:36.211519 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:48:36.211492 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35789086_fdba_4f26_9d20_68b3b6acb945.slice/crio-113b304882d37550fec383ffa6037959b2a9442f90d1c5c75d3af32efd5f72c0 WatchSource:0}: Error finding container 113b304882d37550fec383ffa6037959b2a9442f90d1c5c75d3af32efd5f72c0: Status 404 returned error can't find the container with id 113b304882d37550fec383ffa6037959b2a9442f90d1c5c75d3af32efd5f72c0 Oct 29 22:48:36.511532 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:36.511439 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" event={"ID":"35789086-fdba-4f26-9d20-68b3b6acb945","Type":"ContainerStarted","Data":"113b304882d37550fec383ffa6037959b2a9442f90d1c5c75d3af32efd5f72c0"} Oct 29 22:48:37.518660 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:37.518620 4952 generic.go:356] "Generic (PLEG): container finished" podID="35789086-fdba-4f26-9d20-68b3b6acb945" containerID="96a6406349d4615d3237e32e155269b4b7f8d2cdf42ce2c0507e9d994670dc2c" exitCode=0 Oct 29 22:48:37.519128 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:37.518707 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" event={"ID":"35789086-fdba-4f26-9d20-68b3b6acb945","Type":"ContainerDied","Data":"96a6406349d4615d3237e32e155269b4b7f8d2cdf42ce2c0507e9d994670dc2c"} Oct 29 22:48:38.690753 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:38.690720 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:38.797506 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:38.797408 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35789086-fdba-4f26-9d20-68b3b6acb945-kube-api-access-ds9x2\" (UniqueName: \"kubernetes.io/projected/35789086-fdba-4f26-9d20-68b3b6acb945-kube-api-access-ds9x2\") pod \"35789086-fdba-4f26-9d20-68b3b6acb945\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " Oct 29 22:48:38.797966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:38.797503 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-openshift-ca\") pod \"35789086-fdba-4f26-9d20-68b3b6acb945\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " Oct 29 22:48:38.797966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:38.797606 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-xks-ca\") pod \"35789086-fdba-4f26-9d20-68b3b6acb945\" (UID: \"35789086-fdba-4f26-9d20-68b3b6acb945\") " Oct 29 22:48:38.797966 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:38.797828 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-openshift-ca" pod "35789086-fdba-4f26-9d20-68b3b6acb945" (UID: "35789086-fdba-4f26-9d20-68b3b6acb945"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:38.798153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:38.798007 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-openshift-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:38.799839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:38.799809 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-xks-ca" pod "35789086-fdba-4f26-9d20-68b3b6acb945" (UID: "35789086-fdba-4f26-9d20-68b3b6acb945"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:38.799839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:38.799811 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35789086-fdba-4f26-9d20-68b3b6acb945-kube-api-access-ds9x2" pod "35789086-fdba-4f26-9d20-68b3b6acb945" (UID: "35789086-fdba-4f26-9d20-68b3b6acb945"). InnerVolumeSpecName "kube-api-access-ds9x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:38.899269 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:38.899231 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9x2\" (UniqueName: \"kubernetes.io/projected/35789086-fdba-4f26-9d20-68b3b6acb945-kube-api-access-ds9x2\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:38.899269 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:38.899266 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35789086-fdba-4f26-9d20-68b3b6acb945-maas-api-xks-ca\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:39.532818 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:39.532783 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" Oct 29 22:48:39.533055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:39.532794 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777139-s2g2d" event={"ID":"35789086-fdba-4f26-9d20-68b3b6acb945","Type":"ContainerDied","Data":"113b304882d37550fec383ffa6037959b2a9442f90d1c5c75d3af32efd5f72c0"} Oct 29 22:48:39.533055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:39.532931 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="113b304882d37550fec383ffa6037959b2a9442f90d1c5c75d3af32efd5f72c0" Oct 29 22:48:42.778871 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:42.778796 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" containerID="cri-o://fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0" gracePeriod=45 Oct 29 22:48:42.787141 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:42.787098 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" containerID="cri-o://60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10" gracePeriod=45 Oct 29 22:48:43.076554 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.076520 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 22:48:43.079955 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.079924 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:48:43.146085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146042 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30270cd0-e156-43f1-bab4-ce05b529dca5-kube-api-access-pjkpx\" (UniqueName: \"kubernetes.io/projected/30270cd0-e156-43f1-bab4-ce05b529dca5-kube-api-access-pjkpx\") pod \"30270cd0-e156-43f1-bab4-ce05b529dca5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " Oct 29 22:48:43.146085 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146086 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-model-cache\") pod \"30270cd0-e156-43f1-bab4-ce05b529dca5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " Oct 29 22:48:43.146360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146121 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30270cd0-e156-43f1-bab4-ce05b529dca5-tls-certs\" (UniqueName: \"kubernetes.io/secret/30270cd0-e156-43f1-bab4-ce05b529dca5-tls-certs\") pod \"30270cd0-e156-43f1-bab4-ce05b529dca5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " Oct 29 22:48:43.146360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146149 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-dshm\") pod \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " Oct 29 22:48:43.146360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146166 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-home\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-home\") pod \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " Oct 29 22:48:43.146360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146239 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-tmp-dir\") pod \"30270cd0-e156-43f1-bab4-ce05b529dca5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " Oct 29 22:48:43.146360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146277 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tls-certs\") pod \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " Oct 29 22:48:43.146360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146316 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-kube-api-access-w6rgg\" (UniqueName: \"kubernetes.io/projected/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-kube-api-access-w6rgg\") pod \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " Oct 29 22:48:43.146360 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146348 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-model-cache\") pod \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " Oct 29 22:48:43.146764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146380 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-dshm\") pod \"30270cd0-e156-43f1-bab4-ce05b529dca5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " Oct 29 22:48:43.146764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146419 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-home\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-home\") pod \"30270cd0-e156-43f1-bab4-ce05b529dca5\" (UID: \"30270cd0-e156-43f1-bab4-ce05b529dca5\") " Oct 29 22:48:43.146764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146475 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-model-cache" pod "30270cd0-e156-43f1-bab4-ce05b529dca5" (UID: "30270cd0-e156-43f1-bab4-ce05b529dca5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.146764 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146502 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tmp-dir\") pod \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\" (UID: \"4e9a296f-e4b7-45ad-a57f-3e79b29722ef\") " Oct 29 22:48:43.147066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146777 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-home" pod "4e9a296f-e4b7-45ad-a57f-3e79b29722ef" (UID: "4e9a296f-e4b7-45ad-a57f-3e79b29722ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.147066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146797 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.147066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146810 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-model-cache" pod "4e9a296f-e4b7-45ad-a57f-3e79b29722ef" (UID: "4e9a296f-e4b7-45ad-a57f-3e79b29722ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.147066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.146983 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-tmp-dir" pod "30270cd0-e156-43f1-bab4-ce05b529dca5" (UID: "30270cd0-e156-43f1-bab4-ce05b529dca5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.147066 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.147017 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tmp-dir" pod "4e9a296f-e4b7-45ad-a57f-3e79b29722ef" (UID: "4e9a296f-e4b7-45ad-a57f-3e79b29722ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.147361 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.147251 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-home" pod "30270cd0-e156-43f1-bab4-ce05b529dca5" (UID: "30270cd0-e156-43f1-bab4-ce05b529dca5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.148935 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.148869 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30270cd0-e156-43f1-bab4-ce05b529dca5-tls-certs" pod "30270cd0-e156-43f1-bab4-ce05b529dca5" (UID: "30270cd0-e156-43f1-bab4-ce05b529dca5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:43.149285 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.149259 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30270cd0-e156-43f1-bab4-ce05b529dca5-kube-api-access-pjkpx" pod "30270cd0-e156-43f1-bab4-ce05b529dca5" (UID: "30270cd0-e156-43f1-bab4-ce05b529dca5"). InnerVolumeSpecName "kube-api-access-pjkpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:43.149418 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.149330 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tls-certs" pod "4e9a296f-e4b7-45ad-a57f-3e79b29722ef" (UID: "4e9a296f-e4b7-45ad-a57f-3e79b29722ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:43.149418 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.149370 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-dshm" pod "30270cd0-e156-43f1-bab4-ce05b529dca5" (UID: "30270cd0-e156-43f1-bab4-ce05b529dca5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.149760 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.149728 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-kube-api-access-w6rgg" pod "4e9a296f-e4b7-45ad-a57f-3e79b29722ef" (UID: "4e9a296f-e4b7-45ad-a57f-3e79b29722ef"). InnerVolumeSpecName "kube-api-access-w6rgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:43.150610 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.150593 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-dshm" pod "4e9a296f-e4b7-45ad-a57f-3e79b29722ef" (UID: "4e9a296f-e4b7-45ad-a57f-3e79b29722ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:43.248276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248231 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30270cd0-e156-43f1-bab4-ce05b529dca5-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.248276 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248274 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.248513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248289 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.248513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248303 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.248513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248315 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tls-certs\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.248513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248328 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6rgg\" (UniqueName: \"kubernetes.io/projected/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-kube-api-access-w6rgg\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.248513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248342 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-model-cache\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.248513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248356 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-dshm\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.248513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248367 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30270cd0-e156-43f1-bab4-ce05b529dca5-home\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.248513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248379 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9a296f-e4b7-45ad-a57f-3e79b29722ef-tmp-dir\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.248513 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.248392 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjkpx\" (UniqueName: \"kubernetes.io/projected/30270cd0-e156-43f1-bab4-ce05b529dca5-kube-api-access-pjkpx\") on node \"ip-10-0-130-134.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.560382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.560341 4952 generic.go:356] "Generic (PLEG): container finished" podID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerID="60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10" exitCode=0 Oct 29 22:48:43.560382 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.560382 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" Oct 29 22:48:43.560662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.560433 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" event={"ID":"4e9a296f-e4b7-45ad-a57f-3e79b29722ef","Type":"ContainerDied","Data":"60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10"} Oct 29 22:48:43.560662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.560474 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq" event={"ID":"4e9a296f-e4b7-45ad-a57f-3e79b29722ef","Type":"ContainerDied","Data":"11d1fde1b2d5cec8b0fb15fe6286608c3ac05e74e468780a15255afc22c38120"} Oct 29 22:48:43.560662 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.560496 4952 scope.go:122] "RemoveContainer" containerID="60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10" Oct 29 22:48:43.562168 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.562141 4952 generic.go:356] "Generic (PLEG): container finished" podID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerID="fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0" exitCode=0 Oct 29 22:48:43.562353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.562199 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" event={"ID":"30270cd0-e156-43f1-bab4-ce05b529dca5","Type":"ContainerDied","Data":"fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0"} Oct 29 22:48:43.562353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.562229 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" event={"ID":"30270cd0-e156-43f1-bab4-ce05b529dca5","Type":"ContainerDied","Data":"c69b6f301b821db2a89f1ab04bf9a2b1b6885f50b024f790506375d32fc4afa3"} Oct 29 22:48:43.562353 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.562258 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5" Oct 29 22:48:43.572411 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.572223 4952 scope.go:122] "RemoveContainer" containerID="60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10" Oct 29 22:48:43.572567 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:48:43.572547 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10\": container with ID starting with 60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10 not found: ID does not exist" containerID="60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10" Oct 29 22:48:43.572617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.572577 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10"} err="failed to get container status \"60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10\": rpc error: code = NotFound desc = could not find container \"60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10\": container with ID starting with 60f1b3459f79adbb6f5d24dffe15bca9bafcaeae4a545669ea1c90dd0b967b10 not found: ID does not exist" Oct 29 22:48:43.572617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.572596 4952 scope.go:122] "RemoveContainer" containerID="fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0" Oct 29 22:48:43.581980 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.581959 4952 scope.go:122] "RemoveContainer" containerID="fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0" Oct 29 22:48:43.582267 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:48:43.582249 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0\": container with ID starting with fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0 not found: ID does not exist" containerID="fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0" Oct 29 22:48:43.582309 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.582277 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0"} err="failed to get container status \"fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0\": rpc error: code = NotFound desc = could not find container \"fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0\": container with ID starting with fd8256bbde394109ecae43dde7a6030d684ec0bb8a3de12bde47fc5732db5af0 not found: ID does not exist" Oct 29 22:48:43.618713 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.618680 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5"] Oct 29 22:48:43.631853 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.631816 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/premium-w5-91520c-kserve-67bd66c8bf-xksf5"] Oct 29 22:48:43.663142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.663107 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq"] Oct 29 22:48:43.670108 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:43.670079 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-91520c/simulator-w5-91520c-kserve-55fc57dfb4-k8hcq"] Oct 29 22:48:44.642311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:44.642278 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" path="/var/lib/kubelet/pods/30270cd0-e156-43f1-bab4-ce05b529dca5/volumes" Oct 29 22:48:44.642712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:44.642676 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" path="/var/lib/kubelet/pods/4e9a296f-e4b7-45ad-a57f-3e79b29722ef/volumes" Oct 29 22:48:45.527385 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:48:45.527346 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:48:48.590920 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:48.590870 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:48:50.830956 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:50.830898 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:51.273606 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:51.273504 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:51.443409 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:51.443376 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:51.959982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:51.959940 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:57.592512 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:57.592410 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7gmcc"] Oct 29 22:48:59.343260 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:59.343220 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:59.518566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:59.518519 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:59.712676 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:48:59.712577 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:00.791518 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:00.791482 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68889bc875-jstq6_7b578808-74eb-41d9-8d5a-cdf40c192710/authorino/0.log" Oct 29 22:49:01.025586 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:01.025547 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-khzn6_c071ca3b-2106-4523-a0a5-25f28f2a5a16/manager/0.log" Oct 29 22:49:01.238695 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:01.238661 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-qhdzd_e3de2a0e-ffae-4747-b6d8-03e2d601e991/manager/0.log" Oct 29 22:49:01.917281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:01.917247 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hj67h_68024384-a51d-40b9-bb47-d4e2baac27d1/manager/1.log" Oct 29 22:49:02.236500 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:02.236418 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7gmcc_dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63/limitador/0.log" Oct 29 22:49:02.437247 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:02.437211 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hfj97_df08f7e5-a162-4fb7-8896-4d77f606e4ce/manager/0.log" Oct 29 22:49:03.036566 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:03.036516 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hj67h_68024384-a51d-40b9-bb47-d4e2baac27d1/manager/0.log" Oct 29 22:49:03.898571 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:03.898534 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hwxzn_1ea74cd9-058e-4f28-b2b5-b64902ca8d75/manager/0.log" Oct 29 22:49:04.331097 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:04.331065 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-x2lkg_ebc631c9-7d1c-450d-ad81-114e2ae2e2b2/manager/0.log" Oct 29 22:49:05.132669 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:05.132625 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/1.log" Oct 29 22:49:05.320173 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:05.320137 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pkknp_4cc442ee-a458-43af-84a6-51d7fa97c5bd/server/0.log" Oct 29 22:49:05.533292 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:05.533189 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bk2cb_f33f40ca-9443-4c79-a820-3c5fbad96529/manager/0.log" Oct 29 22:49:06.080160 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:06.080123 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:06.194166 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:06.194129 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/0.log" Oct 29 22:49:06.328684 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:06.328640 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:06.806437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:06.806394 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68889bc875-jstq6_7b578808-74eb-41d9-8d5a-cdf40c192710/authorino/0.log" Oct 29 22:49:07.280017 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:07.279973 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:13.670963 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:13.670890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:14.574828 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:14.574787 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:15.526739 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:49:15.526696 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:49:17.922479 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:17.922445 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hwxzn_1ea74cd9-058e-4f28-b2b5-b64902ca8d75/manager/0.log" Oct 29 22:49:18.353756 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:18.353702 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-x2lkg_ebc631c9-7d1c-450d-ad81-114e2ae2e2b2/manager/0.log" Oct 29 22:49:19.110588 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:19.110547 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/1.log" Oct 29 22:49:19.315270 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:19.315239 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pkknp_4cc442ee-a458-43af-84a6-51d7fa97c5bd/server/0.log" Oct 29 22:49:19.516766 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:19.516648 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bk2cb_f33f40ca-9443-4c79-a820-3c5fbad96529/manager/0.log" Oct 29 22:49:20.189153 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:20.189114 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/0.log" Oct 29 22:49:22.309851 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:22.309818 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c6777139-s2g2d_35789086-fdba-4f26-9d20-68b3b6acb945/revoke-keys/0.log" Oct 29 22:49:24.365786 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:24.365742 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68889bc875-jstq6_7b578808-74eb-41d9-8d5a-cdf40c192710/authorino/0.log" Oct 29 22:49:24.599802 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:24.599766 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-khzn6_c071ca3b-2106-4523-a0a5-25f28f2a5a16/manager/0.log" Oct 29 22:49:24.799832 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:24.799735 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-qhdzd_e3de2a0e-ffae-4747-b6d8-03e2d601e991/manager/0.log" Oct 29 22:49:25.294469 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:25.294428 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hj67h_68024384-a51d-40b9-bb47-d4e2baac27d1/manager/1.log" Oct 29 22:49:25.524940 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:25.524896 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7gmcc_dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63/limitador/0.log" Oct 29 22:49:25.731867 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:25.731835 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hfj97_df08f7e5-a162-4fb7-8896-4d77f606e4ce/manager/0.log" Oct 29 22:49:26.243607 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:26.243570 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hj67h_68024384-a51d-40b9-bb47-d4e2baac27d1/manager/0.log" Oct 29 22:49:26.724741 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:26.724709 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:26.827127 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:26.827097 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c5762925_94cbbe37-02d8-47d2-bc1f-264c9384afeb/istio-proxy/0.log" Oct 29 22:49:26.898027 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:26.897988 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:27.644492 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:27.644462 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ftwkd_ebc99afd-f631-46a3-8d94-bfaae865a10a/istio-proxy/1.log" Oct 29 22:49:28.038982 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:28.038870 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz_b897f2be-4c03-4441-8121-409547c4818d/istio-proxy/0.log" Oct 29 22:49:28.645521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:28.645488 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56bd78965c-kscnl_059a06d7-5255-47dc-8ce7-07d113e03ca1/router/0.log" Oct 29 22:49:29.064490 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:29.064403 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ftwkd_ebc99afd-f631-46a3-8d94-bfaae865a10a/istio-proxy/0.log" Oct 29 22:49:41.197236 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:41.197190 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:41.381239 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:41.381199 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:45.525804 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:49:45.525761 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:49:47.407800 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:47.407743 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-29g9k_fb5b0077-ddac-46d3-ba06-5fc065be5b53/global-pull-secret-syncer/0.log" Oct 29 22:49:47.947018 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:47.946989 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-42pjx_3398371c-c328-4876-963c-8af098bae440/connection-checker/0.log" Oct 29 22:49:48.062022 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:48.061993 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-b425c_3dd58b31-fa23-4a8c-b113-ad9479a7b681/connection-checker/0.log" Oct 29 22:49:48.176746 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:48.176718 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mvw7m_f04d5d43-128a-4972-b55e-836175e60130/connection-checker/0.log" Oct 29 22:49:48.387824 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:48.387790 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" Oct 29 22:49:48.930145 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:48.930113 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-134.ec2.internal_3439c72aff40b3d1f617a398c69856e9/haproxy/0.log" Oct 29 22:49:57.302339 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:57.302301 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68889bc875-jstq6_7b578808-74eb-41d9-8d5a-cdf40c192710/authorino/0.log" Oct 29 22:49:57.403747 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:57.403713 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-khzn6_c071ca3b-2106-4523-a0a5-25f28f2a5a16/manager/0.log" Oct 29 22:49:57.520514 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:57.520484 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-qhdzd_e3de2a0e-ffae-4747-b6d8-03e2d601e991/manager/0.log" Oct 29 22:49:57.891837 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:57.891790 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hj67h_68024384-a51d-40b9-bb47-d4e2baac27d1/manager/0.log" Oct 29 22:49:57.926142 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:57.926090 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hj67h_68024384-a51d-40b9-bb47-d4e2baac27d1/manager/1.log" Oct 29 22:49:59.002743 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.002701 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572"] Oct 29 22:49:59.003307 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003286 4952 cpu_manager.go:414] "containerMap: removing container" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" Oct 29 22:49:59.003307 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003307 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" Oct 29 22:49:59.003447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003349 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" Oct 29 22:49:59.003447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003358 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" Oct 29 22:49:59.003447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003368 4952 cpu_manager.go:414] "containerMap: removing container" podUID="35789086-fdba-4f26-9d20-68b3b6acb945" containerName="revoke-keys" Oct 29 22:49:59.003447 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003377 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35789086-fdba-4f26-9d20-68b3b6acb945" containerName="revoke-keys" Oct 29 22:49:59.003692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003459 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6862987c-7402-4134-a7a7-4a1663aed0de" containerName="istio-proxy" Oct 29 22:49:59.003692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003472 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9a296f-e4b7-45ad-a57f-3e79b29722ef" containerName="main" Oct 29 22:49:59.003692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003479 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="35789086-fdba-4f26-9d20-68b3b6acb945" containerName="revoke-keys" Oct 29 22:49:59.003692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003493 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="345f5596-f75f-4aa1-adc4-d97f16ffa1a0" containerName="istio-proxy" Oct 29 22:49:59.003692 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.003503 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="30270cd0-e156-43f1-bab4-ce05b529dca5" containerName="main" Oct 29 22:49:59.007356 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.007334 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.028046 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.028011 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572"] Oct 29 22:49:59.050574 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.050541 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-podres\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.050754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.050585 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-lib-modules\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.050754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.050623 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-sys\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.050754 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.050715 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qvg\" (UniqueName: \"kubernetes.io/projected/6a8649a6-92fc-48b6-b4de-98a982fedf87-kube-api-access-d6qvg\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.050877 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.050822 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-proc\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.060344 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.060315 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gbwp5\"/\"default-dockercfg-ldfvl\"" Oct 29 22:49:59.111627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.111594 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gbwp5\"/\"kube-root-ca.crt\"" Oct 29 22:49:59.111627 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.111604 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gbwp5\"/\"openshift-service-ca.crt\"" Oct 29 22:49:59.152245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.152192 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-podres\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.152245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.152244 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-lib-modules\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.152489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.152290 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-sys\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.152489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.152313 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qvg\" (UniqueName: \"kubernetes.io/projected/6a8649a6-92fc-48b6-b4de-98a982fedf87-kube-api-access-d6qvg\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.152489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.152359 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-podres\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.152489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.152366 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-proc\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.152489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.152415 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-proc\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.152489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.152422 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-lib-modules\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.152489 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.152415 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a8649a6-92fc-48b6-b4de-98a982fedf87-sys\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.181595 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.181546 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qvg\" (UniqueName: \"kubernetes.io/projected/6a8649a6-92fc-48b6-b4de-98a982fedf87-kube-api-access-d6qvg\") pod \"perf-node-gather-daemonset-kq572\" (UID: \"6a8649a6-92fc-48b6-b4de-98a982fedf87\") " pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.203657 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.203629 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7gmcc_dc8e9b1d-d4ab-4ea4-96a6-7247aa4e2f63/limitador/0.log" Oct 29 22:49:59.281784 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.281701 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hwxzn_1ea74cd9-058e-4f28-b2b5-b64902ca8d75/manager/0.log" Oct 29 22:49:59.320427 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.320386 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:49:59.419245 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.419182 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-hfj97_df08f7e5-a162-4fb7-8896-4d77f606e4ce/manager/0.log" Oct 29 22:49:59.550281 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.550191 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hwxzn_1ea74cd9-058e-4f28-b2b5-b64902ca8d75/manager/0.log" Oct 29 22:49:59.645206 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:49:59.645163 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572"] Oct 29 22:49:59.653683 ip-10-0-130-134 kubenswrapper[4952]: W1029 22:49:59.653649 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6a8649a6_92fc_48b6_b4de_98a982fedf87.slice/crio-ad7df652aef67fed425c30c2f9688f3194dfceb86cfa0be82ec85ea819fc9a47 WatchSource:0}: Error finding container ad7df652aef67fed425c30c2f9688f3194dfceb86cfa0be82ec85ea819fc9a47: Status 404 returned error can't find the container with id ad7df652aef67fed425c30c2f9688f3194dfceb86cfa0be82ec85ea819fc9a47 Oct 29 22:50:00.009055 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:00.009009 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" event={"ID":"6a8649a6-92fc-48b6-b4de-98a982fedf87","Type":"ContainerStarted","Data":"1d6815ee8d1416599216b1e60a3640a17066455e90620c41ae04fc43fa8f4878"} Oct 29 22:50:00.009494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:00.009084 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" event={"ID":"6a8649a6-92fc-48b6-b4de-98a982fedf87","Type":"ContainerStarted","Data":"ad7df652aef67fed425c30c2f9688f3194dfceb86cfa0be82ec85ea819fc9a47"} Oct 29 22:50:00.009494 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:00.009116 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:50:00.055148 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:00.055089 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" podStartSLOduration=2.055071229 podStartE2EDuration="2.055071229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:00.053623726 +0000 UTC m=+4065.528212353" watchObservedRunningTime="2026-10-29 22:50:00.055071229 +0000 UTC m=+4065.529659649" Oct 29 22:50:00.871314 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:00.871273 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-x2lkg_ebc631c9-7d1c-450d-ad81-114e2ae2e2b2/manager/0.log" Oct 29 22:50:00.917927 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:00.917877 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/0.log" Oct 29 22:50:01.289712 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:01.289611 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-x2lkg_ebc631c9-7d1c-450d-ad81-114e2ae2e2b2/manager/0.log" Oct 29 22:50:01.349045 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:01.349007 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/0.log" Oct 29 22:50:01.700099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:01.700058 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/1.log" Oct 29 22:50:01.735539 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:01.735507 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pkknp_4cc442ee-a458-43af-84a6-51d7fa97c5bd/server/0.log" Oct 29 22:50:01.753891 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:01.753850 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-m4ks6_33291806-5f11-4f13-ba59-92a143fe8f0e/cluster-monitoring-operator/0.log" Oct 29 22:50:02.150167 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:02.150125 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/1.log" Oct 29 22:50:02.188628 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:02.188593 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pkknp_4cc442ee-a458-43af-84a6-51d7fa97c5bd/server/0.log" Oct 29 22:50:02.528991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:02.528823 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bk2cb_f33f40ca-9443-4c79-a820-3c5fbad96529/manager/0.log" Oct 29 22:50:02.609057 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:02.609005 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:02.795123 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:02.795026 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:02.888664 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:02.888611 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bk2cb_f33f40ca-9443-4c79-a820-3c5fbad96529/manager/0.log" Oct 29 22:50:03.246020 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:03.245989 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ft2d5_9c5d8661-d9da-45e0-b9d5-4a83096a3985/node-exporter/0.log" Oct 29 22:50:03.352791 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:03.352759 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ft2d5_9c5d8661-d9da-45e0-b9d5-4a83096a3985/kube-rbac-proxy/0.log" Oct 29 22:50:03.460617 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:03.460586 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ft2d5_9c5d8661-d9da-45e0-b9d5-4a83096a3985/init-textfile/0.log" Oct 29 22:50:04.159219 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:04.159187 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hwxzn_1ea74cd9-058e-4f28-b2b5-b64902ca8d75/manager/0.log" Oct 29 22:50:04.525711 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:04.525614 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hwxzn_1ea74cd9-058e-4f28-b2b5-b64902ca8d75/manager/0.log" Oct 29 22:50:05.691641 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:05.691593 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-x2lkg_ebc631c9-7d1c-450d-ad81-114e2ae2e2b2/manager/0.log" Oct 29 22:50:05.767562 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:05.767531 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/0.log" Oct 29 22:50:06.029334 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:06.029240 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gbwp5/perf-node-gather-daemonset-kq572" Oct 29 22:50:06.252517 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:06.252475 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-x2lkg_ebc631c9-7d1c-450d-ad81-114e2ae2e2b2/manager/0.log" Oct 29 22:50:06.326269 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:06.326227 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/0.log" Oct 29 22:50:06.574099 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:06.574058 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/1.log" Oct 29 22:50:06.713135 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:06.713046 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pkknp_4cc442ee-a458-43af-84a6-51d7fa97c5bd/server/0.log" Oct 29 22:50:07.029856 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:07.029755 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/1.log" Oct 29 22:50:07.120298 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:07.120266 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pkknp_4cc442ee-a458-43af-84a6-51d7fa97c5bd/server/0.log" Oct 29 22:50:07.723794 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:07.723753 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bk2cb_f33f40ca-9443-4c79-a820-3c5fbad96529/manager/0.log" Oct 29 22:50:07.909184 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:07.909147 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bk2cb_f33f40ca-9443-4c79-a820-3c5fbad96529/manager/0.log" Oct 29 22:50:07.939198 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:07.939170 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-zv2wb_1e889f05-b993-4423-9b9c-4673ad6e70d3/networking-console-plugin/0.log" Oct 29 22:50:09.567042 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:09.567004 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/2.log" Oct 29 22:50:09.575593 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:09.575563 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7jz5d_20070f80-4b65-40ff-9646-e8b7fef85cc1/console-operator/3.log" Oct 29 22:50:09.898311 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:09.898278 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hwxzn_1ea74cd9-058e-4f28-b2b5-b64902ca8d75/manager/0.log" Oct 29 22:50:09.922410 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:09.922371 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hwxzn_1ea74cd9-058e-4f28-b2b5-b64902ca8d75/manager/0.log" Oct 29 22:50:11.629329 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:11.629273 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-x2lkg_ebc631c9-7d1c-450d-ad81-114e2ae2e2b2/manager/0.log" Oct 29 22:50:11.658359 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:11.658314 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-x2lkg_ebc631c9-7d1c-450d-ad81-114e2ae2e2b2/manager/0.log" Oct 29 22:50:11.811699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:11.811651 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/0.log" Oct 29 22:50:11.817060 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:11.816986 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/0.log" Oct 29 22:50:12.324229 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:12.324191 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-jtzgk_f8539056-ee01-4262-89b4-1d4b55a5073c/volume-data-source-validator/0.log" Oct 29 22:50:12.604636 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:12.604533 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/1.log" Oct 29 22:50:12.607215 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:12.607185 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8bdd4cf6-g6mpt_bdefc3de-7cff-41cd-8cbc-d2dd14ef1e4f/manager/1.log" Oct 29 22:50:12.790884 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:12.790856 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pkknp_4cc442ee-a458-43af-84a6-51d7fa97c5bd/server/0.log" Oct 29 22:50:12.797437 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:12.797410 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pkknp_4cc442ee-a458-43af-84a6-51d7fa97c5bd/server/0.log" Oct 29 22:50:13.630888 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:13.630847 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bk2cb_f33f40ca-9443-4c79-a820-3c5fbad96529/manager/0.log" Oct 29 22:50:13.635675 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:13.635640 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bk2cb_f33f40ca-9443-4c79-a820-3c5fbad96529/manager/0.log" Oct 29 22:50:14.123461 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:14.123429 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c5762925_94cbbe37-02d8-47d2-bc1f-264c9384afeb/istio-proxy/0.log" Oct 29 22:50:14.762835 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:14.762765 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-krcw7_73219cee-7208-4f68-ad0b-bc9cb0bcb199/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:15.123420 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:15.123386 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mmxmn_b39d65b2-cca6-4f08-bdec-becd6e000c48/dns/0.log" Oct 29 22:50:15.268637 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:15.268607 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mmxmn_b39d65b2-cca6-4f08-bdec-becd6e000c48/kube-rbac-proxy/0.log" Oct 29 22:50:15.529469 ip-10-0-130-134 kubenswrapper[4952]: E1029 22:50:15.529391 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8dsqf" containerName="extractor" Oct 29 22:50:15.758699 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:15.758664 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c5762925_94cbbe37-02d8-47d2-bc1f-264c9384afeb/istio-proxy/0.log" Oct 29 22:50:15.790289 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:15.790190 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c5762925_94cbbe37-02d8-47d2-bc1f-264c9384afeb/istio-proxy/0.log" Oct 29 22:50:16.151271 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:16.151234 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t6k4p_84058be2-867e-4c5a-a614-bccfd7af96fa/dns-node-resolver/0.log" Oct 29 22:50:16.375926 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:16.375859 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ftwkd_ebc99afd-f631-46a3-8d94-bfaae865a10a/istio-proxy/0.log" Oct 29 22:50:16.376433 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:16.376238 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ftwkd_ebc99afd-f631-46a3-8d94-bfaae865a10a/istio-proxy/1.log" Oct 29 22:50:16.490779 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:16.490662 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5579cb49c9-2bkzz_b897f2be-4c03-4441-8121-409547c4818d/istio-proxy/0.log" Oct 29 22:50:16.713991 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:16.713940 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56bd78965c-kscnl_059a06d7-5255-47dc-8ce7-07d113e03ca1/router/0.log" Oct 29 22:50:17.725839 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:17.725803 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tjgwl_89de847b-bcca-4f80-af6b-071681d3afdb/node-ca/0.log" Oct 29 22:50:18.825355 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:18.825308 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ftwkd_ebc99afd-f631-46a3-8d94-bfaae865a10a/istio-proxy/0.log" Oct 29 22:50:18.826521 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:18.826486 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ftwkd_ebc99afd-f631-46a3-8d94-bfaae865a10a/istio-proxy/1.log" Oct 29 22:50:18.864264 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:18.864224 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ftwkd_ebc99afd-f631-46a3-8d94-bfaae865a10a/istio-proxy/0.log" Oct 29 22:50:18.867218 ip-10-0-130-134 kubenswrapper[4952]: I1029 22:50:18.867191 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ftwkd_ebc99afd-f631-46a3-8d94-bfaae865a10a/istio-proxy/1.log"